[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:02 dist-test-slave-th6x krb5kdc[3779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:02 dist-test-slave-th6x krb5kdc[3779](info): set up 2 sockets
Oct 26 21:06:02 dist-test-slave-th6x krb5kdc[3779](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251026 21:06:04.856743 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:04 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:04.999938 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44927
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:44927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:05.114286 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:05.114578 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:05.114645 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:05.118116 3795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:05.118194 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:05.118218 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:05.118238 3795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:05.118256 3795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:05.122861 3795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:44927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:05.124130 3795 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:05.125051 3795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 21:06:05.131834 3795 server_base.cc:1047] running on GCE node
W20251026 21:06:05.131819 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.131762 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.131769 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:05.132606 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:05.133736 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:05.134915 3795 hybrid_clock.cc:648] HybridClock initialized: now 1761512765134905 us; error 33 us; skew 500 ppm
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:05.137986 3795 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:05.139106 3795 webserver.cc:492] Webserver started at http://127.3.175.62:46843/ using document root <none> and password file <none>
I20251026 21:06:05.139717 3795 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:05.139793 3795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:05.140059 3795 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:05.141805 3795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "c2d7fbd0bf274379b37a0a03c26089da"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "95e88c528d6d3d413da2a2c920cb8302"
server_key_iv: "60e88bcf98f3e64754654e9cb715cf56"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.142290 3795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "c2d7fbd0bf274379b37a0a03c26089da"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "95e88c528d6d3d413da2a2c920cb8302"
server_key_iv: "60e88bcf98f3e64754654e9cb715cf56"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.145995 3795 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:05.148520 3810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.149772 3795 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:05.149940 3795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "c2d7fbd0bf274379b37a0a03c26089da"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "95e88c528d6d3d413da2a2c920cb8302"
server_key_iv: "60e88bcf98f3e64754654e9cb715cf56"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.150053 3795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:05.202438 3795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:05.206396 3795 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:05.206640 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:05.214370 3795 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:44927
I20251026 21:06:05.214365 3862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:44927 every 8 connection(s)
I20251026 21:06:05.215556 3795 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:05.217105 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 3795
I20251026 21:06:05.217245 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:05.218238 3772 external_mini_cluster.cc:1468] Setting key bfc2a678a747176b178888e30ae1a928
I20251026 21:06:05.218647 3863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:05.225544 3863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da: Bootstrap starting.
I20251026 21:06:05.228070 3863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:05.228884 3863 log.cc:826] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:05.231185 3863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da: No bootstrap required, opened a new log
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:05.234391 3863 raft_consensus.cc:359] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } }
I20251026 21:06:05.234659 3863 raft_consensus.cc:385] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:05.234748 3863 raft_consensus.cc:740] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d7fbd0bf274379b37a0a03c26089da, State: Initialized, Role: FOLLOWER
I20251026 21:06:05.235265 3863 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } }
I20251026 21:06:05.235455 3863 raft_consensus.cc:399] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:05.235530 3863 raft_consensus.cc:493] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:05.235634 3863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:05.236795 3863 raft_consensus.cc:515] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } }
I20251026 21:06:05.237195 3863 leader_election.cc:304] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d7fbd0bf274379b37a0a03c26089da; no voters:
I20251026 21:06:05.237563 3863 leader_election.cc:290] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:05.237658 3868 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:05.237846 3868 raft_consensus.cc:697] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [term 1 LEADER]: Becoming Leader. State: Replica: c2d7fbd0bf274379b37a0a03c26089da, State: Running, Role: LEADER
I20251026 21:06:05.238170 3868 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } }
I20251026 21:06:05.238852 3863 sys_catalog.cc:565] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:05.239151 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:05.220767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.3.175.62:44927)
1026 21:06:05.221298 (+ 531us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:05.221313 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:05.221505 (+ 192us) server_negotiation.cc:408] Connection header received
1026 21:06:05.224211 (+ 2706us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:05.224239 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:05.224646 (+ 407us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:05.225005 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:05.227009 (+ 2004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.228175 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:05.228975 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.229239 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:05.233179 (+ 3940us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:05.233207 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:05.233221 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:05.233301 (+ 80us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:05.235695 (+ 2394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.236259 (+ 564us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.236266 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.236271 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.236373 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.236757 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.236762 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.236767 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.237239 (+ 472us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:05.237459 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:05.238169 (+ 710us) server_negotiation.cc:300] Negotiation successful
1026 21:06:05.238439 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":131,"threads_started":1}
I20251026 21:06:05.239439 3868 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d7fbd0bf274379b37a0a03c26089da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } } }
I20251026 21:06:05.239555 3868 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:05.239925 3872 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:05.240104 3869 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d7fbd0bf274379b37a0a03c26089da. Latest consensus state: current_term: 1 leader_uuid: "c2d7fbd0bf274379b37a0a03c26089da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d7fbd0bf274379b37a0a03c26089da" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44927 } } }
I20251026 21:06:05.240433 3869 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:05.243204 3872 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:05.248757 3872 catalog_manager.cc:1357] Generated new cluster ID: 407effd9e5e34ebf991b5b8bb45d031e
I20251026 21:06:05.248844 3872 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:05.264292 3872 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:05.265290 3872 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:05.271458 3872 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c2d7fbd0bf274379b37a0a03c26089da: Generated new TSK 0
I20251026 21:06:05.272332 3872 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:05.324651 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44927
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:05.455202 3891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:05.455503 3891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:05.455581 3891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:05.458961 3891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:05.459033 3891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:05.459144 3891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:05.463544 3891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:05.464715 3891 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:05.465607 3891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:05.472515 3896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.472504 3897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.472501 3899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:05.472839 3891 server_base.cc:1047] running on GCE node
I20251026 21:06:05.473229 3891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:05.473822 3891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:05.475005 3891 hybrid_clock.cc:648] HybridClock initialized: now 1761512765474981 us; error 40 us; skew 500 ppm
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:05.477921 3891 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:05.479053 3891 webserver.cc:492] Webserver started at http://127.3.175.1:36467/ using document root <none> and password file <none>
I20251026 21:06:05.479666 3891 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:05.479722 3891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:05.479890 3891 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:05.481739 3891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "ec3371592bfa4685a65e898b84e1fc65"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d30a5cfbd2a3179fd76732ff94e89127"
server_key_iv: "0bf76197ea93adc147f548ca2f41f3e8"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.482204 3891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "ec3371592bfa4685a65e898b84e1fc65"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d30a5cfbd2a3179fd76732ff94e89127"
server_key_iv: "0bf76197ea93adc147f548ca2f41f3e8"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.485805 3891 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:05.488221 3906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.489255 3891 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:05.489354 3891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "ec3371592bfa4685a65e898b84e1fc65"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d30a5cfbd2a3179fd76732ff94e89127"
server_key_iv: "0bf76197ea93adc147f548ca2f41f3e8"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.489439 3891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:05.507736 3891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:05.511013 3891 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:05.511183 3891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:05.511798 3891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:05.512756 3891 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:05.512807 3891 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.512851 3891 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:05.512890 3891 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.521876 3891 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:34691
I20251026 21:06:05.521893 4019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:34691 every 8 connection(s)
I20251026 21:06:05.522899 3891 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:05.532130 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 3891
I20251026 21:06:05.532244 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:05.532526 3772 external_mini_cluster.cc:1468] Setting key f92076d1f8893db5fd4d18d5bec2bb0d
I20251026 21:06:05.536865 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:05.524748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:43685 (local address 127.3.175.62:44927)
1026 21:06:05.524938 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:05.524944 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:05.525770 (+ 826us) server_negotiation.cc:408] Connection header received
1026 21:06:05.526692 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:05.526695 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:05.526750 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:05.526818 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:05.528714 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.529245 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:05.529939 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.530068 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:05.532512 (+ 2444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:05.532539 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:05.532541 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:05.532569 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:05.534390 (+ 1821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.535074 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.535077 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.535078 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.535126 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.535484 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.535486 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.535488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.535655 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:05.535737 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:05.536466 (+ 729us) server_negotiation.cc:300] Negotiation successful
1026 21:06:05.536659 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251026 21:06:05.537546 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:05.525032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44927 (local address 127.3.175.1:43685)
1026 21:06:05.525600 (+ 568us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:05.525644 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:05.526457 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:05.527044 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:05.527053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:05.527520 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:05.528536 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:05.528556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.529393 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:05.529397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:05.529843 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:05.529851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.530048 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:05.530673 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:05.530699 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:05.532331 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:05.534509 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:05.534516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:05.534531 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:05.534957 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:05.535213 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:05.535219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:05.535224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:05.535396 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:05.535773 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:05.535784 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:05.536153 (+ 369us) client_negotiation.cc:769] Sending connection context
1026 21:06:05.536348 (+ 195us) client_negotiation.cc:240] Negotiation successful
1026 21:06:05.536605 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":115,"threads_started":1}
I20251026 21:06:05.539140 4020 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44927
I20251026 21:06:05.539498 4020 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:05.540298 4020 heartbeater.cc:507] Master 127.3.175.62:44927 requested a full tablet report, sending...
I20251026 21:06:05.542063 3827 ts_manager.cc:194] Registered new tserver with Master: ec3371592bfa4685a65e898b84e1fc65 (127.3.175.1:34691)
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:06:05.543345 3827 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:43685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:05.586110 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44927
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:05.702924 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:05.703166 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:05.703226 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:05.706670 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:05.706751 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:05.706840 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:05.711416 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:05.712558 4027 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:05.713411 4027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:05.720165 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.720242 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.720165 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:05.720635 4027 server_base.cc:1047] running on GCE node
I20251026 21:06:05.721033 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:05.721612 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:05.722797 4027 hybrid_clock.cc:648] HybridClock initialized: now 1761512765722779 us; error 42 us; skew 500 ppm
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:05.725636 4027 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:05.727003 4027 webserver.cc:492] Webserver started at http://127.3.175.2:43007/ using document root <none> and password file <none>
I20251026 21:06:05.727665 4027 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:05.727725 4027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:05.727916 4027 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:05.729694 4027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "5027c084fefe4d10b18aa4b0ba325e9b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d0cb3fe86b468993b86b547ff5dcea5d"
server_key_iv: "6a272fcf8e78afe211ac8a0f1cdf96ac"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.730227 4027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "5027c084fefe4d10b18aa4b0ba325e9b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d0cb3fe86b468993b86b547ff5dcea5d"
server_key_iv: "6a272fcf8e78afe211ac8a0f1cdf96ac"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.733938 4027 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251026 21:06:05.736372 4042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.737442 4027 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:05.737565 4027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "5027c084fefe4d10b18aa4b0ba325e9b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "d0cb3fe86b468993b86b547ff5dcea5d"
server_key_iv: "6a272fcf8e78afe211ac8a0f1cdf96ac"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.737661 4027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:05.775847 4027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:05.779090 4027 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:05.779320 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:05.779937 4027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:05.780870 4027 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:05.780923 4027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.780987 4027 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:05.781016 4027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.790741 4027 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:38369
I20251026 21:06:05.790787 4155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:38369 every 8 connection(s)
I20251026 21:06:05.791791 4027 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:05.792726 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4027
I20251026 21:06:05.792853 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:05.793107 3772 external_mini_cluster.cc:1468] Setting key fae115c2416ca3b992417e55dff6c077
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:05.806694 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:05.793930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:44499 (local address 127.3.175.62:44927)
1026 21:06:05.794087 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:05.794092 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:05.795021 (+ 929us) server_negotiation.cc:408] Connection header received
1026 21:06:05.795962 (+ 941us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:05.795969 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:05.796041 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:05.796141 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:05.798360 (+ 2219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.799264 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:05.800048 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.800271 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:05.802855 (+ 2584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:05.802878 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:05.802881 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:05.802911 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:05.804518 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.805159 (+ 641us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.805165 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.805170 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.805235 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:05.805556 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:05.805559 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:05.805561 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:05.805740 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:05.805869 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:05.806374 (+ 505us) server_negotiation.cc:300] Negotiation successful
1026 21:06:05.806511 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 21:06:05.807549 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:05.794294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44927 (local address 127.3.175.2:44499)
1026 21:06:05.794854 (+ 560us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:05.794899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:05.795741 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:05.796363 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:05.796379 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:05.797028 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:05.798177 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:05.798199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.799468 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:05.799472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:05.799910 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:05.799918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:05.800128 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:05.800808 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:05.800836 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:05.802661 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:05.804671 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:05.804677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:05.804688 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:05.805038 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:05.805343 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:05.805346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:05.805347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:05.805449 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:05.805850 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:05.805857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:05.806122 (+ 265us) client_negotiation.cc:769] Sending connection context
1026 21:06:05.806327 (+ 205us) client_negotiation.cc:240] Negotiation successful
1026 21:06:05.806671 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":175,"threads_started":1}
I20251026 21:06:05.808745 4156 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44927
I20251026 21:06:05.809054 4156 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:05.809777 4156 heartbeater.cc:507] Master 127.3.175.62:44927 requested a full tablet report, sending...
I20251026 21:06:05.811053 3827 ts_manager.cc:194] Registered new tserver with Master: 5027c084fefe4d10b18aa4b0ba325e9b (127.3.175.2:38369)
I20251026 21:06:05.811734 3827 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:44499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:05.846616 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44927
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:05.959492 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:05.959748 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:05.959834 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:05.963243 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:05.963384 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:05.963517 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:05.967957 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:05.969146 4163 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:05.970032 4163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:05.977775 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.977818 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:05.977900 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:05.978013 4163 server_base.cc:1047] running on GCE node
I20251026 21:06:05.978463 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:05.979053 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:05.980208 4163 hybrid_clock.cc:648] HybridClock initialized: now 1761512765980191 us; error 39 us; skew 500 ppm
Oct 26 21:06:05 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:05.985994 4163 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:05.987143 4163 webserver.cc:492] Webserver started at http://127.3.175.3:40871/ using document root <none> and password file <none>
I20251026 21:06:05.987738 4163 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:05.987792 4163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:05.987953 4163 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:05.989665 4163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "5957c6d164574e48b6b17696fdf0ee8b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "b8702c6768cf38bdff77d16533cc91f8"
server_key_iv: "67919012c7f6f73dde62f3126561f243"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.990125 4163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "5957c6d164574e48b6b17696fdf0ee8b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "b8702c6768cf38bdff77d16533cc91f8"
server_key_iv: "67919012c7f6f73dde62f3126561f243"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.993722 4163 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251026 21:06:05.996248 4178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:05.997414 4163 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:06:05.997606 4163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "5957c6d164574e48b6b17696fdf0ee8b"
format_stamp: "Formatted at 2025-10-26 21:06:05 on dist-test-slave-th6x"
server_key: "b8702c6768cf38bdff77d16533cc91f8"
server_key_iv: "67919012c7f6f73dde62f3126561f243"
server_key_version: "encryptionkey@0"
I20251026 21:06:05.997721 4163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:06.021098 4163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:06.024681 4163 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:06.024875 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:06.025477 4163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:06.026448 4163 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:06.026502 4163 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:06.026576 4163 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:06.026607 4163 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:06.035868 4163 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:39557
I20251026 21:06:06.035903 4291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:39557 every 8 connection(s)
I20251026 21:06:06.036871 4163 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:06.043426 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4163
I20251026 21:06:06.043533 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:06.043782 3772 external_mini_cluster.cc:1468] Setting key 925a064d42e51297d55dfb4f19e6bbd2
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:06.050328 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:06.038696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:41273 (local address 127.3.175.62:44927)
1026 21:06:06.038837 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:06.038841 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:06.039604 (+ 763us) server_negotiation.cc:408] Connection header received
1026 21:06:06.040496 (+ 892us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:06.040499 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:06.040593 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:06.040669 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:06.042441 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.043040 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:06.043772 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.043954 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:06.046782 (+ 2828us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:06.046800 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:06.046802 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:06.046848 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:06.048367 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.048929 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.048933 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.048934 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.048978 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.049247 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.049250 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.049252 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.049409 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:06.049500 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:06.050032 (+ 532us) server_negotiation.cc:300] Negotiation successful
1026 21:06:06.050153 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:06:06.050995 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:06.038956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44927 (local address 127.3.175.3:41273)
1026 21:06:06.039419 (+ 463us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:06.039455 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:06.040290 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:06.040848 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:06.040856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:06.041259 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:06.042276 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:06.042296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.043168 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:06.043171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:06.043658 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:06.043670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.043915 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:06.044673 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:06.044710 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:06.046629 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:06.048503 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:06.048509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:06.048523 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:06.048825 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:06.049063 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:06.049068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:06.049071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:06.049161 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:06.049513 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:06.049522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:06.049781 (+ 259us) client_negotiation.cc:769] Sending connection context
1026 21:06:06.049986 (+ 205us) client_negotiation.cc:240] Negotiation successful
1026 21:06:06.050268 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20251026 21:06:06.052179 4292 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44927
I20251026 21:06:06.052408 4292 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:06.052939 4292 heartbeater.cc:507] Master 127.3.175.62:44927 requested a full tablet report, sending...
I20251026 21:06:06.054054 3827 ts_manager.cc:194] Registered new tserver with Master: 5957c6d164574e48b6b17696fdf0ee8b (127.3.175.3:39557)
I20251026 21:06:06.054706 3827 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:41273
I20251026 21:06:06.058298 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:06.083187 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:06.074121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.3.175.1:34691)
1026 21:06:06.074449 (+ 328us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:06.074456 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:06.074479 (+ 23us) server_negotiation.cc:408] Connection header received
1026 21:06:06.074539 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:06.074544 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:06.074715 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:06.074843 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:06.075812 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.076562 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:06.077244 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.077434 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:06.079773 (+ 2339us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:06.079819 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:06.079832 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:06.079860 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:06.081553 (+ 1693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.082010 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.082015 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.082017 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.082074 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.082361 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.082364 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.082365 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.082597 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:06.082786 (+ 189us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:06.082923 (+ 137us) server_negotiation.cc:300] Negotiation successful
1026 21:06:06.083053 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
W20251026 21:06:06.084198 3934 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53308
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[3779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[3779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:06.108814 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:06.100752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.3.175.1:34691)
1026 21:06:06.100946 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:06.100950 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:06.100966 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:06:06.101085 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:06.101088 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:06.101139 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:06.101212 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:06.102096 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.102605 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:06.103357 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:06.103529 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:06.105821 (+ 2292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:06.105843 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:06.105845 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:06.105875 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:06.107356 (+ 1481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.107792 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.107795 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.107797 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.107845 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:06.108044 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:06.108047 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:06.108048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:06.108190 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:06.108301 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:06.108567 (+ 266us) server_negotiation.cc:300] Negotiation successful
1026 21:06:06.108673 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20251026 21:06:06.109879 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 3891
I20251026 21:06:06.116225 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4027
I20251026 21:06:06.124339 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4163
I20251026 21:06:06.130267 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 3795
2025-10-26T21:06:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3286 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[4306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[4306](info): set up 2 sockets
Oct 26 21:06:06 dist-test-slave-th6x krb5kdc[4306](info): commencing operation
krb5kdc: starting...
W20251026 21:06:08.143674 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/test-user.keytab.
W20251026 21:06:08.495942 3772 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.221s user 0.000s sys 0.002s
Oct 26 21:06:08 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
W20251026 21:06:09.417717 3772 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.3.175.62: real 0.901s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
W20251026 21:06:10.819868 3772 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.3.175.62: real 1.402s user 0.000s sys 0.004s
I20251026 21:06:10.820407 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:36219
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:10.942305 4322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:10.942615 4322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:10.942682 4322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:10.946347 4322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:10.946439 4322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:10.946463 4322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:10.946482 4322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:10.946501 4322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:10.951599 4322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:10.952935 4322 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:10.953934 4322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:10.960233 4327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:10.960260 4328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:10.960233 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:10.960598 4322 server_base.cc:1047] running on GCE node
I20251026 21:06:10.961203 4322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:10.962303 4322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:10.963505 4322 hybrid_clock.cc:648] HybridClock initialized: now 1761512770963471 us; error 61 us; skew 500 ppm
Oct 26 21:06:10 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:10.966724 4322 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:10.968003 4322 webserver.cc:492] Webserver started at http://127.3.175.62:46393/ using document root <none> and password file <none>
I20251026 21:06:10.968605 4322 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:10.968662 4322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:10.968861 4322 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:10.970770 4322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "ae84c805419c4e9bbe5b4f75a66e27e3"
format_stamp: "Formatted at 2025-10-26 21:06:10 on dist-test-slave-th6x"
server_key: "450d6db7b100ec40c5b8f33e1ccb8cfb"
server_key_iv: "f87d2268bb8248a4b723485ed8a27efb"
server_key_version: "encryptionkey@0"
I20251026 21:06:10.971342 4322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "ae84c805419c4e9bbe5b4f75a66e27e3"
format_stamp: "Formatted at 2025-10-26 21:06:10 on dist-test-slave-th6x"
server_key: "450d6db7b100ec40c5b8f33e1ccb8cfb"
server_key_iv: "f87d2268bb8248a4b723485ed8a27efb"
server_key_version: "encryptionkey@0"
I20251026 21:06:10.975314 4322 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20251026 21:06:10.978068 4337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:10.979485 4322 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:10.979643 4322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "ae84c805419c4e9bbe5b4f75a66e27e3"
format_stamp: "Formatted at 2025-10-26 21:06:10 on dist-test-slave-th6x"
server_key: "450d6db7b100ec40c5b8f33e1ccb8cfb"
server_key_iv: "f87d2268bb8248a4b723485ed8a27efb"
server_key_version: "encryptionkey@0"
I20251026 21:06:10.979761 4322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:11.002929 4322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:11.029119 4322 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:11.029323 4322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:11.037446 4322 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:36219
I20251026 21:06:11.037501 4389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:36219 every 8 connection(s)
I20251026 21:06:11.038625 4322 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:11.041860 4390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:11.047935 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4322
I20251026 21:06:11.048069 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:11.048309 3772 external_mini_cluster.cc:1468] Setting key 6f27479d9b2ac66aef92d91436e1a6d1
I20251026 21:06:11.048497 4390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: Bootstrap starting.
I20251026 21:06:11.051324 4390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:11.052171 4390 log.cc:826] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:11.054661 4390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: No bootstrap required, opened a new log
Oct 26 21:06:11 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:11.058167 4390 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } }
I20251026 21:06:11.058449 4390 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:11.058516 4390 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae84c805419c4e9bbe5b4f75a66e27e3, State: Initialized, Role: FOLLOWER
I20251026 21:06:11.059118 4390 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } }
I20251026 21:06:11.059316 4390 raft_consensus.cc:399] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:11.059407 4390 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:11.059530 4390 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:11.064136 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:11.049860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.3.175.62:36219)
1026 21:06:11.050369 (+ 509us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:11.050380 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:11.050419 (+ 39us) server_negotiation.cc:408] Connection header received
1026 21:06:11.051089 (+ 670us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:11.051110 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:11.051565 (+ 455us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:11.051953 (+ 388us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:11.053090 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:11.054301 (+ 1211us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:11.055157 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:11.055475 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:11.057949 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:11.057979 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:11.057994 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:11.058054 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:11.060535 (+ 2481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:11.061098 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:11.061104 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:11.061109 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:11.061206 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:11.061466 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:11.061469 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:11.061471 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:11.061941 (+ 470us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:11.062137 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:11.062503 (+ 366us) server_negotiation.cc:300] Negotiation successful
1026 21:06:11.062781 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":166,"threads_started":1}
I20251026 21:06:11.122238 4390 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } }
I20251026 21:06:11.122820 4390 leader_election.cc:304] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae84c805419c4e9bbe5b4f75a66e27e3; no voters:
I20251026 21:06:11.123198 4390 leader_election.cc:290] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:11.123391 4395 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:11.123656 4395 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [term 1 LEADER]: Becoming Leader. State: Replica: ae84c805419c4e9bbe5b4f75a66e27e3, State: Running, Role: LEADER
I20251026 21:06:11.124740 4390 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:11.124850 4395 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } }
I20251026 21:06:11.126868 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } } }
I20251026 21:06:11.126886 4397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae84c805419c4e9bbe5b4f75a66e27e3. Latest consensus state: current_term: 1 leader_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae84c805419c4e9bbe5b4f75a66e27e3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36219 } } }
I20251026 21:06:11.127045 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:11.127046 4397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3 [sys.catalog]: This master's current role is: LEADER
W20251026 21:06:11.130087 4409 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 21:06:11.130177 4409 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 21:06:11.130300 4410 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:11.131502 4410 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:11.138185 4410 catalog_manager.cc:1357] Generated new cluster ID: f16e0d63bf6a45a4bbed847aba0f2112
I20251026 21:06:11.138293 4410 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:11.159579 4410 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:11.161099 4410 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:11.170583 4410 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ae84c805419c4e9bbe5b4f75a66e27e3: Generated new TSK 0
I20251026 21:06:11.171701 4410 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
W20251026 21:06:11.955394 3772 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.3.175.1: real 0.773s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:11.983496 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36219
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:12.104058 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:12.104315 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:12.104382 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:12.107908 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:12.107996 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:12.108104 4418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:12.112840 4418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:12.114077 4418 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:12.115025 4418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:12.122473 4426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:12.122592 4418 server_base.cc:1047] running on GCE node
W20251026 21:06:12.122466 4424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:12.122466 4423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:12.123239 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:12.123924 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:12.125080 4418 hybrid_clock.cc:648] HybridClock initialized: now 1761512772125065 us; error 30 us; skew 500 ppm
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:12.128146 4418 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:12.129338 4418 webserver.cc:492] Webserver started at http://127.3.175.1:43409/ using document root <none> and password file <none>
I20251026 21:06:12.129920 4418 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:12.129973 4418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:12.130162 4418 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:12.132011 4418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "6938ca749bbe4b95b25dc725222d9845"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "750b57cdc4ae01bb407795fe26c32f4b"
server_key_iv: "8ac0844f0343f59394b06aaeec40413a"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.132543 4418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "6938ca749bbe4b95b25dc725222d9845"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "750b57cdc4ae01bb407795fe26c32f4b"
server_key_iv: "8ac0844f0343f59394b06aaeec40413a"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.136277 4418 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:12.138772 4433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.140066 4418 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:12.140201 4418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "6938ca749bbe4b95b25dc725222d9845"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "750b57cdc4ae01bb407795fe26c32f4b"
server_key_iv: "8ac0844f0343f59394b06aaeec40413a"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.140311 4418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:12.156911 4418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:12.160503 4418 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:12.160712 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:12.161449 4418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:12.162488 4418 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:12.162544 4418 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.162617 4418 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:12.162652 4418 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.172576 4418 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:36119
I20251026 21:06:12.172585 4546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:36119 every 8 connection(s)
I20251026 21:06:12.173725 4418 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:12.179952 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4418
I20251026 21:06:12.180083 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:12.180321 3772 external_mini_cluster.cc:1468] Setting key 5f217de7ee842b916a5dbfd40ce90561
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:12.189113 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.175892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:50133 (local address 127.3.175.62:36219)
1026 21:06:12.176157 (+ 265us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.176160 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.176691 (+ 531us) server_negotiation.cc:408] Connection header received
1026 21:06:12.177618 (+ 927us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.177623 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.177690 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.177777 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.179656 (+ 1879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.180444 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.182036 (+ 1592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.182273 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.184742 (+ 2469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.184764 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.184766 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.184794 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.186732 (+ 1938us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.187344 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.187350 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.187355 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.187428 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.187779 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.187784 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.187788 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.188044 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.188172 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.188726 (+ 554us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.188892 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20251026 21:06:12.189689 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.176061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36219 (local address 127.3.175.1:50133)
1026 21:06:12.176528 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.176568 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.177389 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.177974 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.177983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.178417 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:12.179481 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.179498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.180686 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.180690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.181326 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.181340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.181664 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.182462 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:12.182494 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:12.184559 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:12.186895 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.186901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.186913 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.187192 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.187535 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.187538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.187540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.187676 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.188160 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:12.188166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:12.188435 (+ 269us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.188648 (+ 213us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.188915 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:06:12.191056 4547 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36219
I20251026 21:06:12.191368 4547 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:12.191970 4547 heartbeater.cc:507] Master 127.3.175.62:36219 requested a full tablet report, sending...
I20251026 21:06:12.193805 4348 ts_manager.cc:194] Registered new tserver with Master: 6938ca749bbe4b95b25dc725222d9845 (127.3.175.1:36119)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:12.195588 4348 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:50133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:12.238986 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36219
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:12.353350 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:12.353633 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:12.353698 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:12.357204 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:12.357292 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:12.357401 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:12.361985 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:12.363166 4555 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:12.364130 4555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:12.371039 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:12.371305 4555 server_base.cc:1047] running on GCE node
W20251026 21:06:12.371039 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:12.371047 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:12.371853 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:12.372496 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:12.373705 4555 hybrid_clock.cc:648] HybridClock initialized: now 1761512772373688 us; error 34 us; skew 500 ppm
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:12.376761 4555 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:12.378006 4555 webserver.cc:492] Webserver started at http://127.3.175.2:41815/ using document root <none> and password file <none>
I20251026 21:06:12.378593 4555 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:12.378654 4555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:12.378857 4555 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:12.380713 4555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "6ef066de69594d2497b74e2b34763b09"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "8c4472fb982098682e065b323e1577f8"
server_key_iv: "2e6a9762d5449a3398cb951a4e8769cb"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.381237 4555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "6ef066de69594d2497b74e2b34763b09"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "8c4472fb982098682e065b323e1577f8"
server_key_iv: "2e6a9762d5449a3398cb951a4e8769cb"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.385241 4555 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251026 21:06:12.387801 4570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.388975 4555 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:12.389107 4555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "6ef066de69594d2497b74e2b34763b09"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "8c4472fb982098682e065b323e1577f8"
server_key_iv: "2e6a9762d5449a3398cb951a4e8769cb"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.389218 4555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:12.407454 4555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:12.410720 4555 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:12.410953 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:12.411585 4555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:12.412526 4555 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:12.412580 4555 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.412644 4555 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:12.412683 4555 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.422230 4555 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:42035
I20251026 21:06:12.422307 4683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:42035 every 8 connection(s)
I20251026 21:06:12.423532 4555 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:12.425518 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4555
I20251026 21:06:12.425647 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:12.425908 3772 external_mini_cluster.cc:1468] Setting key a66e58d1b20ab242042c7118143f5dd2
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:12.437585 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.425456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:58845 (local address 127.3.175.62:36219)
1026 21:06:12.425690 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.425698 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.426426 (+ 728us) server_negotiation.cc:408] Connection header received
1026 21:06:12.427394 (+ 968us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.427398 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.427454 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.427546 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.429310 (+ 1764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.430131 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.430883 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.431120 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.433662 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.433694 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.433700 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.433741 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.435484 (+ 1743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.436069 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.436073 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.436074 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.436123 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.436421 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.436423 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.436424 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.436578 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.436668 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.437258 (+ 590us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.437382 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:12.438231 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.425785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36219 (local address 127.3.175.2:58845)
1026 21:06:12.426279 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.426327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.427180 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.427724 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.427733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.428170 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:12.429148 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.429163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.430278 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.430281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.430752 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.430761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.430957 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.431686 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:12.431716 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:12.433475 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:12.435612 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.435618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.435629 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.435918 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.436221 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.436223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.436225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.436344 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.436668 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:12.436674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:12.436939 (+ 265us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.437169 (+ 230us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.437418 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20251026 21:06:12.439458 4684 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36219
I20251026 21:06:12.439751 4684 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:12.440308 4684 heartbeater.cc:507] Master 127.3.175.62:36219 requested a full tablet report, sending...
I20251026 21:06:12.441547 4348 ts_manager.cc:194] Registered new tserver with Master: 6ef066de69594d2497b74e2b34763b09 (127.3.175.2:42035)
I20251026 21:06:12.442448 4348 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:58845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:12.479871 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36219
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:12.592533 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:12.592792 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:12.592854 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:12.596271 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:12.596345 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:12.596433 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:12.600943 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:12.602094 4691 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:12.602939 4691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:12.609844 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:12.609844 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:12.610054 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:12.610121 4691 server_base.cc:1047] running on GCE node
I20251026 21:06:12.610555 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:12.611156 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:12.612334 4691 hybrid_clock.cc:648] HybridClock initialized: now 1761512772612321 us; error 30 us; skew 500 ppm
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:12.615195 4691 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:12.616390 4691 webserver.cc:492] Webserver started at http://127.3.175.3:33637/ using document root <none> and password file <none>
I20251026 21:06:12.616962 4691 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:12.617012 4691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:12.617175 4691 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:12.619313 4691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "d699cb6dbc574a41893c635705f2d28e"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "6b16c67cb93850262c0c1ea36b241e4b"
server_key_iv: "47fd07985522fa73871e644a27c27ec5"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.619836 4691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "d699cb6dbc574a41893c635705f2d28e"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "6b16c67cb93850262c0c1ea36b241e4b"
server_key_iv: "47fd07985522fa73871e644a27c27ec5"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.623548 4691 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:12.626209 4706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.627661 4691 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:12.627796 4691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "d699cb6dbc574a41893c635705f2d28e"
format_stamp: "Formatted at 2025-10-26 21:06:12 on dist-test-slave-th6x"
server_key: "6b16c67cb93850262c0c1ea36b241e4b"
server_key_iv: "47fd07985522fa73871e644a27c27ec5"
server_key_version: "encryptionkey@0"
I20251026 21:06:12.627894 4691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:12.653396 4691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:12.657006 4691 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:12.657200 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:12.657764 4691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:12.658689 4691 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:12.658740 4691 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.658782 4691 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:12.658797 4691 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:12.668543 4691 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:35553
I20251026 21:06:12.668619 4819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:35553 every 8 connection(s)
I20251026 21:06:12.669623 4691 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:12.676512 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4691
I20251026 21:06:12.676666 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:12.676932 3772 external_mini_cluster.cc:1468] Setting key 413cec5693127a0c06263489410e3461
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:12.683959 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.671516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:57503 (local address 127.3.175.62:36219)
1026 21:06:12.671668 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.671672 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.672449 (+ 777us) server_negotiation.cc:408] Connection header received
1026 21:06:12.673344 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.673348 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.673414 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.673492 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.675336 (+ 1844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.675854 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.676727 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.676930 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.680007 (+ 3077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.680023 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.680025 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.680056 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.681574 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.682206 (+ 632us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.682209 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.682210 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.682261 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.682713 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.682717 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.682718 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.682919 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.683016 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.683690 (+ 674us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.683818 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:06:12.684713 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.671770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36219 (local address 127.3.175.3:57503)
1026 21:06:12.672284 (+ 514us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.672322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.673130 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.673683 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.673692 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.674142 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:12.675131 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.675147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.675985 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.675988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.676506 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.676516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.676823 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.677719 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:12.677768 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:12.679807 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:12.681768 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.681774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.681786 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.682091 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.682385 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:12.682388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:12.682389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:12.682502 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:12.683073 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:12.683079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:12.683403 (+ 324us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.683636 (+ 233us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.683905 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20251026 21:06:12.685933 4820 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36219
I20251026 21:06:12.686184 4820 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:12.686749 4820 heartbeater.cc:507] Master 127.3.175.62:36219 requested a full tablet report, sending...
I20251026 21:06:12.687903 4348 ts_manager.cc:194] Registered new tserver with Master: d699cb6dbc574a41893c635705f2d28e (127.3.175.3:35553)
I20251026 21:06:12.688453 4348 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:57503
I20251026 21:06:12.691598 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:06:12.701586 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.694693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.3.175.62:36219)
1026 21:06:12.694838 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.694841 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.695033 (+ 192us) server_negotiation.cc:408] Connection header received
1026 21:06:12.695160 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.695163 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.695210 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.695275 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.696170 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.696699 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.697366 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.697503 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.698653 (+ 1150us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.698673 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.698676 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.698699 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.700249 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.700636 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.700638 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.700640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.700678 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.700867 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.700868 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.700870 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.701073 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.701180 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.701340 (+ 160us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.701457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:06:12.706117 4348 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:12.708669 4348 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:12.723392 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.718422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:35553 (local address 127.0.0.1:44946)
1026 21:06:12.718874 (+ 452us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.718914 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.719064 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.720399 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.720405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.720445 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:12.720794 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.720804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.721909 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.721914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.722770 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.722780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.722887 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.722970 (+ 83us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.723079 (+ 109us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.723193 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"spinlock_wait_cycles":8832,"thread_start_us":140,"threads_started":1}
I20251026 21:06:12.723907 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.718913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:36119 (local address 127.0.0.1:40962)
1026 21:06:12.719583 (+ 670us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.719600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.719690 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.720399 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.720405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.720443 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:12.720779 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.720791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.722209 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.722213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.723139 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.723149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.723232 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.723250 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.723328 (+ 78us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.723407 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"spinlock_wait_cycles":6016,"thread_start_us":97,"threads_started":1}
I20251026 21:06:12.724140 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.718602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.3.175.3:35553)
1026 21:06:12.719984 (+ 1382us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.719990 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.720010 (+ 20us) server_negotiation.cc:408] Connection header received
1026 21:06:12.720084 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.720088 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.720228 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.720652 (+ 424us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:12.720957 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.721743 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.723102 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.723716 (+ 614us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.723824 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.723896 (+ 72us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.723977 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":147,"threads_started":1}
I20251026 21:06:12.724620 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.719236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:42035 (local address 127.0.0.1:49656)
1026 21:06:12.719723 (+ 487us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.719740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.719836 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.720821 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.720824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.720846 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:12.721084 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.721091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.722475 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.722479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.723327 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.723335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.723441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.723460 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.723513 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.724499 (+ 986us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":109,"threads_started":1}
I20251026 21:06:12.724814 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.3.175.2:42035)
1026 21:06:12.719937 (+ 493us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.719945 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.719964 (+ 19us) server_negotiation.cc:408] Connection header received
1026 21:06:12.720030 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.720034 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.720217 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.720371 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:12.721209 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.722353 (+ 1144us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.723828 (+ 1475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.724381 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.724497 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.724575 (+ 78us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.724680 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":120,"threads_started":1}
I20251026 21:06:12.725064 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.719371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.3.175.1:36119)
1026 21:06:12.719885 (+ 514us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.719893 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.719916 (+ 23us) server_negotiation.cc:408] Connection header received
1026 21:06:12.720012 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.720018 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.720217 (+ 199us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.720371 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:12.720968 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.722069 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.724126 (+ 2057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.724651 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.724750 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.724817 (+ 67us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.724930 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":111,"threads_started":1}
I20251026 21:06:12.727571 4481 tablet_service.cc:1505] Processing CreateTablet for tablet c176b96ec1e4474698242ab902377de2 (DEFAULT_TABLE table=test-table [id=201333219469431abef7ad6921daa5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:12.727571 4618 tablet_service.cc:1505] Processing CreateTablet for tablet c176b96ec1e4474698242ab902377de2 (DEFAULT_TABLE table=test-table [id=201333219469431abef7ad6921daa5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:12.727586 4752 tablet_service.cc:1505] Processing CreateTablet for tablet c176b96ec1e4474698242ab902377de2 (DEFAULT_TABLE table=test-table [id=201333219469431abef7ad6921daa5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:12.728538 4481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176b96ec1e4474698242ab902377de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:12.728540 4618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176b96ec1e4474698242ab902377de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:12.728540 4752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176b96ec1e4474698242ab902377de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:12.734537 4838 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Bootstrap starting.
I20251026 21:06:12.736017 4839 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Bootstrap starting.
I20251026 21:06:12.736124 4840 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Bootstrap starting.
I20251026 21:06:12.737969 4840 tablet_bootstrap.cc:654] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:12.737999 4838 tablet_bootstrap.cc:654] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:12.737999 4839 tablet_bootstrap.cc:654] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:12.738766 4840 log.cc:826] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:12.738826 4838 log.cc:826] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:12.738826 4839 log.cc:826] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:12.740751 4838 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: No bootstrap required, opened a new log
I20251026 21:06:12.740749 4839 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: No bootstrap required, opened a new log
I20251026 21:06:12.740979 4839 ts_tablet_manager.cc:1403] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:06:12.740963 4838 ts_tablet_manager.cc:1403] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251026 21:06:12.742225 4840 tablet_bootstrap.cc:492] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: No bootstrap required, opened a new log
I20251026 21:06:12.742478 4840 ts_tablet_manager.cc:1403] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251026 21:06:12.743889 4838 raft_consensus.cc:359] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.744104 4838 raft_consensus.cc:385] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:12.744182 4838 raft_consensus.cc:740] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6938ca749bbe4b95b25dc725222d9845, State: Initialized, Role: FOLLOWER
I20251026 21:06:12.744629 4838 consensus_queue.cc:260] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.745007 4839 raft_consensus.cc:359] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.745296 4839 raft_consensus.cc:385] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:12.745373 4839 raft_consensus.cc:740] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef066de69594d2497b74e2b34763b09, State: Initialized, Role: FOLLOWER
I20251026 21:06:12.745579 4547 heartbeater.cc:499] Master 127.3.175.62:36219 was elected leader, sending a full tablet report...
I20251026 21:06:12.745708 4838 ts_tablet_manager.cc:1434] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251026 21:06:12.745934 4839 consensus_queue.cc:260] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.746384 4840 raft_consensus.cc:359] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.746591 4840 raft_consensus.cc:385] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:12.746639 4840 raft_consensus.cc:740] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d699cb6dbc574a41893c635705f2d28e, State: Initialized, Role: FOLLOWER
I20251026 21:06:12.746668 4684 heartbeater.cc:499] Master 127.3.175.62:36219 was elected leader, sending a full tablet report...
I20251026 21:06:12.746951 4839 ts_tablet_manager.cc:1434] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20251026 21:06:12.747099 4840 consensus_queue.cc:260] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.748044 4840 ts_tablet_manager.cc:1434] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20251026 21:06:12.748060 4820 heartbeater.cc:499] Master 127.3.175.62:36219 was elected leader, sending a full tablet report...
W20251026 21:06:12.920607 4821 tablet.cc:2378] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:12.924749 4685 tablet.cc:2378] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:12.925076 4548 tablet.cc:2378] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:12.932637 4844 raft_consensus.cc:493] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:12.932826 4844 raft_consensus.cc:515] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.934087 4844 leader_election.cc:290] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d699cb6dbc574a41893c635705f2d28e (127.3.175.3:35553), 6ef066de69594d2497b74e2b34763b09 (127.3.175.2:42035)
I20251026 21:06:12.938460 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.934351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:35553 (local address 127.3.175.1:37177)
1026 21:06:12.934713 (+ 362us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.934731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.934854 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.935175 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.935178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.935192 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:12.935802 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.935810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.936977 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.936981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.937973 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.937983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.938112 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.938135 (+ 23us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.938188 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.938253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":159,"threads_started":1}
I20251026 21:06:12.938460 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.934835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:42035 (local address 127.3.175.1:33457)
1026 21:06:12.935090 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:12.935101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:12.935181 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:12.935584 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:12.935588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:12.935618 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:12.935894 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.935904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.936977 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.936981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:12.937973 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:12.937983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.938112 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.938135 (+ 23us) client_negotiation.cc:769] Sending connection context
1026 21:06:12.938188 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 21:06:12.938253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20251026 21:06:12.938892 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.934997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:33457 (local address 127.3.175.2:42035)
1026 21:06:12.935148 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.935152 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.935166 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:06:12.935419 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.935422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.935473 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.935554 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:12.936032 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.936849 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.938116 (+ 1267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.938617 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.938658 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.938712 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.938771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251026 21:06:12.938892 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.934461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:37177 (local address 127.3.175.3:35553)
1026 21:06:12.934598 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.934603 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.934856 (+ 253us) server_negotiation.cc:408] Connection header received
1026 21:06:12.934983 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.934987 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.935050 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.935155 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:12.935986 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.936849 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.938127 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.938623 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.938658 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.938712 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.938771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251026 21:06:12.939529 4774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176b96ec1e4474698242ab902377de2" candidate_uuid: "6938ca749bbe4b95b25dc725222d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d699cb6dbc574a41893c635705f2d28e" is_pre_election: true
I20251026 21:06:12.939530 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176b96ec1e4474698242ab902377de2" candidate_uuid: "6938ca749bbe4b95b25dc725222d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef066de69594d2497b74e2b34763b09" is_pre_election: true
I20251026 21:06:12.939819 4638 raft_consensus.cc:2468] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6938ca749bbe4b95b25dc725222d9845 in term 0.
I20251026 21:06:12.939817 4774 raft_consensus.cc:2468] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6938ca749bbe4b95b25dc725222d9845 in term 0.
I20251026 21:06:12.940286 4434 leader_election.cc:304] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6938ca749bbe4b95b25dc725222d9845, d699cb6dbc574a41893c635705f2d28e; no voters:
I20251026 21:06:12.940623 4844 raft_consensus.cc:2804] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:12.940701 4844 raft_consensus.cc:493] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:12.940734 4844 raft_consensus.cc:3060] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:12.941754 4844 raft_consensus.cc:515] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.942143 4844 leader_election.cc:290] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [CANDIDATE]: Term 1 election: Requested vote from peers d699cb6dbc574a41893c635705f2d28e (127.3.175.3:35553), 6ef066de69594d2497b74e2b34763b09 (127.3.175.2:42035)
I20251026 21:06:12.942579 4774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176b96ec1e4474698242ab902377de2" candidate_uuid: "6938ca749bbe4b95b25dc725222d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d699cb6dbc574a41893c635705f2d28e"
I20251026 21:06:12.942716 4774 raft_consensus.cc:3060] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:12.942740 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176b96ec1e4474698242ab902377de2" candidate_uuid: "6938ca749bbe4b95b25dc725222d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef066de69594d2497b74e2b34763b09"
I20251026 21:06:12.942878 4638 raft_consensus.cc:3060] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:12.943876 4774 raft_consensus.cc:2468] T c176b96ec1e4474698242ab902377de2 P d699cb6dbc574a41893c635705f2d28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6938ca749bbe4b95b25dc725222d9845 in term 1.
I20251026 21:06:12.943907 4638 raft_consensus.cc:2468] T c176b96ec1e4474698242ab902377de2 P 6ef066de69594d2497b74e2b34763b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6938ca749bbe4b95b25dc725222d9845 in term 1.
I20251026 21:06:12.944226 4434 leader_election.cc:304] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6938ca749bbe4b95b25dc725222d9845, 6ef066de69594d2497b74e2b34763b09; no voters:
I20251026 21:06:12.944432 4844 raft_consensus.cc:2804] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:12.944722 4844 raft_consensus.cc:697] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [term 1 LEADER]: Becoming Leader. State: Replica: 6938ca749bbe4b95b25dc725222d9845, State: Running, Role: LEADER
I20251026 21:06:12.945076 4844 consensus_queue.cc:237] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } }
I20251026 21:06:12.948343 4348 catalog_manager.cc:5649] T c176b96ec1e4474698242ab902377de2 P 6938ca749bbe4b95b25dc725222d9845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6938ca749bbe4b95b25dc725222d9845 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "6938ca749bbe4b95b25dc725222d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d699cb6dbc574a41893c635705f2d28e" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef066de69594d2497b74e2b34763b09" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6938ca749bbe4b95b25dc725222d9845" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36119 } health_report { overall_health: HEALTHY } } }
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:12.965309 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.956226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.3.175.1:36119)
1026 21:06:12.956379 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.956385 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.956465 (+ 80us) server_negotiation.cc:408] Connection header received
1026 21:06:12.956608 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.956611 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.956668 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.956744 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.957711 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.958232 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.958911 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.959140 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.961591 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.961631 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.961644 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.961675 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.963606 (+ 1931us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.964045 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.964051 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.964053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.964111 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.964342 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.964346 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.964348 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.964551 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.964783 (+ 232us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.964994 (+ 211us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.965139 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:12.991077 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:12.983324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.3.175.1:36119)
1026 21:06:12.983499 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:12.983503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:12.983561 (+ 58us) server_negotiation.cc:408] Connection header received
1026 21:06:12.983732 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:12.983736 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:12.983785 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:12.983857 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:12.984727 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.985257 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:12.985908 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:12.986043 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:12.988348 (+ 2305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:12.988374 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:12.988377 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:12.988404 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:12.989778 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.990153 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.990156 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.990157 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.990195 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:12.990344 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:12.990346 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:12.990347 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:12.990481 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:12.990586 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:12.990803 (+ 217us) server_negotiation.cc:300] Negotiation successful
1026 21:06:12.990903 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Oct 26 21:06:12 dist-test-slave-th6x krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 21:06:13 dist-test-slave-th6x krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:13.017087 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:13.009012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.3.175.1:36119)
1026 21:06:13.009156 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:13.009163 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:13.009227 (+ 64us) server_negotiation.cc:408] Connection header received
1026 21:06:13.009406 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:13.009409 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:13.009457 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:13.009525 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:13.010393 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:13.010880 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:13.011604 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:13.011825 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:13.014091 (+ 2266us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:13.014114 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:13.014117 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:13.014149 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:13.015602 (+ 1453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:13.016022 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:13.016025 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:13.016027 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:13.016073 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:13.016286 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:13.016290 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:13.016292 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:13.016439 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:13.016577 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:13.016783 (+ 206us) server_negotiation.cc:300] Negotiation successful
1026 21:06:13.016915 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20251026 21:06:13.018396 4460 tablet_service.cc:3012] Rejecting scan request for tablet c176b96ec1e4474698242ab902377de2: Uninitialized: clean time has not yet been initialized
I20251026 21:06:13.020059 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4418
I20251026 21:06:13.026520 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4555
I20251026 21:06:13.032397 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4691
I20251026 21:06:13.038401 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4322
2025-10-26T21:06:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6908 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:13 dist-test-slave-th6x krb5kdc[4863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:13 dist-test-slave-th6x krb5kdc[4863](info): set up 2 sockets
Oct 26 21:06:13 dist-test-slave-th6x krb5kdc[4863](info): commencing operation
krb5kdc: starting...
W20251026 21:06:15.039942 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:15.186964 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:35963
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:35963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:15.301970 4879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:15.302286 4879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:15.302390 4879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:15.305902 4879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:15.306005 4879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 21:06:15.306044 4879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:15.306084 4879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:15.306131 4879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:15.306178 4879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:15.310889 4879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:35963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:35963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:15.312260 4879 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:15.313235 4879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:15.319387 4884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:15.319511 4887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:15.319700 4879 server_base.cc:1047] running on GCE node
W20251026 21:06:15.319387 4885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:15.320272 4879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:15.321233 4879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:15.322391 4879 hybrid_clock.cc:648] HybridClock initialized: now 1761512775322379 us; error 30 us; skew 500 ppm
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:15.325235 4879 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:15.326400 4879 webserver.cc:492] Webserver started at http://127.3.175.62:42305/ using document root <none> and password file <none>
I20251026 21:06:15.326979 4879 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:15.327034 4879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:15.327242 4879 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:15.329072 4879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "e5c2664154ad43838bb5e38252c2f3e6"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "6a091c079b2158ca6eadfecd8e39489c"
server_key_iv: "c01d108460e03d10110d1c2047eeb0d7"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.329535 4879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "e5c2664154ad43838bb5e38252c2f3e6"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "6a091c079b2158ca6eadfecd8e39489c"
server_key_iv: "c01d108460e03d10110d1c2047eeb0d7"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.333082 4879 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:15.335362 4894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:15.336524 4879 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:15.336643 4879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "e5c2664154ad43838bb5e38252c2f3e6"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "6a091c079b2158ca6eadfecd8e39489c"
server_key_iv: "c01d108460e03d10110d1c2047eeb0d7"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.336745 4879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:15.355247 4879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:15.358433 4879 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:15.358641 4879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:15.367399 4879 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:35963
I20251026 21:06:15.367606 4956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:35963 every 8 connection(s)
I20251026 21:06:15.368496 4879 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:15.371467 4957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:15.373363 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4879
I20251026 21:06:15.373462 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:15.373692 3772 external_mini_cluster.cc:1468] Setting key 4023362db10b72e04487d4e7a41362b6
I20251026 21:06:15.377972 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Bootstrap starting.
I20251026 21:06:15.380524 4957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:15.381276 4957 log.cc:826] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Log is configured to *not* fsync() on all Append() calls
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:15.383762 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: No bootstrap required, opened a new log
I20251026 21:06:15.387362 4957 raft_consensus.cc:359] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } }
I20251026 21:06:15.387598 4957 raft_consensus.cc:385] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:15.387657 4957 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c2664154ad43838bb5e38252c2f3e6, State: Initialized, Role: FOLLOWER
I20251026 21:06:15.387866 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.375023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.3.175.62:35963)
1026 21:06:15.375489 (+ 466us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:15.375501 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:15.375544 (+ 43us) server_negotiation.cc:408] Connection header received
1026 21:06:15.376193 (+ 649us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:15.376224 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:15.376619 (+ 395us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:15.377021 (+ 402us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:15.377805 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.378940 (+ 1135us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.379615 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.379898 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.382231 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:15.382257 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:15.382276 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:15.382325 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:15.384524 (+ 2199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.384986 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.384992 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.384998 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.385097 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.385300 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.385304 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.385306 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.385780 (+ 474us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:15.386499 (+ 719us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:15.386706 (+ 207us) server_negotiation.cc:300] Negotiation successful
1026 21:06:15.387009 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20251026 21:06:15.388164 4957 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } }
I20251026 21:06:15.388295 4957 raft_consensus.cc:399] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:15.388362 4957 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:15.388453 4957 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:15.389536 4957 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } }
I20251026 21:06:15.389938 4957 leader_election.cc:304] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c2664154ad43838bb5e38252c2f3e6; no voters:
I20251026 21:06:15.390285 4957 leader_election.cc:290] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:15.390396 4962 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:15.390612 4962 raft_consensus.cc:697] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [term 1 LEADER]: Becoming Leader. State: Replica: e5c2664154ad43838bb5e38252c2f3e6, State: Running, Role: LEADER
I20251026 21:06:15.390964 4962 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } }
I20251026 21:06:15.391479 4957 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:15.392575 4963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c2664154ad43838bb5e38252c2f3e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } } }
I20251026 21:06:15.392686 4963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:15.393002 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c2664154ad43838bb5e38252c2f3e6. Latest consensus state: current_term: 1 leader_uuid: "e5c2664154ad43838bb5e38252c2f3e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2664154ad43838bb5e38252c2f3e6" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35963 } } }
I20251026 21:06:15.393088 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:15.393158 4971 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:15.396127 4971 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:15.401602 4971 catalog_manager.cc:1357] Generated new cluster ID: 3f8e1e00aedd48f0a44f031ccacf67bf
I20251026 21:06:15.401700 4971 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:15.417073 4971 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:15.418473 4971 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:15.426828 4971 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Generated new TSK 0
I20251026 21:06:15.427862 4971 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:15.490348 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:35963
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:15.607791 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:15.608048 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:15.608114 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:15.611510 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:15.611584 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:15.611647 4985 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:15.611689 4985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:15.616191 4985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:35963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:15.617344 4985 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:15.618208 4985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:15.625455 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:15.625610 4985 server_base.cc:1047] running on GCE node
W20251026 21:06:15.625455 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:15.625455 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:15.626163 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:15.626777 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:15.627929 4985 hybrid_clock.cc:648] HybridClock initialized: now 1761512775627916 us; error 30 us; skew 500 ppm
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:15.630710 4985 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:15.632123 4985 webserver.cc:492] Webserver started at http://127.3.175.1:42195/ using document root <none> and password file <none>
I20251026 21:06:15.632869 4985 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:15.632938 4985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:15.633172 4985 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:15.635459 4985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "037a229cb09aaf9a0b65c4f0806d5d39"
server_key_iv: "8003593a7aab9029fd7db136d312341c"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.636090 4985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "037a229cb09aaf9a0b65c4f0806d5d39"
server_key_iv: "8003593a7aab9029fd7db136d312341c"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.640405 4985 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20251026 21:06:15.642829 5000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:15.643987 4985 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:15.644104 4985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "037a229cb09aaf9a0b65c4f0806d5d39"
server_key_iv: "8003593a7aab9029fd7db136d312341c"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.644212 4985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:15.667518 4985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:15.671522 4985 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:15.671751 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:15.673697 4985 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:15.673763 4985 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:15.673825 4985 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:15.673854 4985 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:15.684924 4985 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:36411
I20251026 21:06:15.684950 5115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:36411 every 8 connection(s)
I20251026 21:06:15.685956 4985 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:15.686618 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.674177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:58963 (local address 127.3.175.62:35963)
1026 21:06:15.674337 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:15.674342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:15.675101 (+ 759us) server_negotiation.cc:408] Connection header received
1026 21:06:15.676046 (+ 945us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:15.676050 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:15.676112 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:15.676223 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:15.677905 (+ 1682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.678779 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.679723 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.679984 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.682594 (+ 2610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:15.682630 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:15.682636 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:15.682679 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:15.684511 (+ 1832us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.685117 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.685121 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.685123 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.685186 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.685475 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.685478 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.685481 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.685661 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:15.685760 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:15.686304 (+ 544us) server_negotiation.cc:300] Negotiation successful
1026 21:06:15.686450 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:06:15.687211 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 4985
I20251026 21:06:15.687332 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:15.687361 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.674496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.1:58963)
1026 21:06:15.674933 (+ 437us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:15.674978 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:15.675830 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:15.676369 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:15.676378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:15.676940 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:15.677729 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.677744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.678927 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.678933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:15.679559 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.679571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.679821 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.680551 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:15.680594 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:15.682407 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:15.684680 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.684685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.684695 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.685009 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.685287 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.685290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.685292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.685382 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.685770 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:15.685779 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:15.686046 (+ 267us) client_negotiation.cc:769] Sending connection context
1026 21:06:15.686250 (+ 204us) client_negotiation.cc:240] Negotiation successful
1026 21:06:15.686533 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20251026 21:06:15.687611 3772 external_mini_cluster.cc:1468] Setting key 295008b69ab085b0214feedaaa477713
I20251026 21:06:15.688691 5116 heartbeater.cc:344] Connected to a master server at 127.3.175.62:35963
I20251026 21:06:15.688851 5116 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:15.689496 5116 heartbeater.cc:507] Master 127.3.175.62:35963 requested a full tablet report, sending...
I20251026 21:06:15.691115 4910 ts_manager.cc:194] Registered new tserver with Master: 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1:36411)
I20251026 21:06:15.692368 4910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:58963
I20251026 21:06:15.697816 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.690850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:48721 (local address 127.3.175.62:35963)
1026 21:06:15.690986 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:15.690992 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:15.691174 (+ 182us) server_negotiation.cc:408] Connection header received
1026 21:06:15.691334 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:15.691338 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:15.691395 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:15.691464 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:15.692162 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.692940 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.693829 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.693973 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.694909 (+ 936us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:15.694927 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:15.694929 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:15.694952 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:15.696382 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.696847 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.696850 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.696851 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.696891 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.697117 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.697118 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.697119 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.697248 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:15.697315 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:15.697591 (+ 276us) server_negotiation.cc:300] Negotiation successful
1026 21:06:15.697694 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:06:15.697937 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.690790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.1:48721)
1026 21:06:15.691096 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:15.691108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:15.691202 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:15.691491 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:15.691494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:15.691723 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:15.692018 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.692030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.693102 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.693106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:15.693657 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.693679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.693831 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.694389 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:15.694405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:15.694782 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:15.696508 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.696515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.696520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.696753 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.696971 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.696977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.696981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.697041 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.697328 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:15.697331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:15.697434 (+ 103us) client_negotiation.cc:769] Sending connection context
1026 21:06:15.697556 (+ 122us) client_negotiation.cc:240] Negotiation successful
1026 21:06:15.697714 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:15.742609 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:35963
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:15.858404 5127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:15.858688 5127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:15.858785 5127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:15.862360 5127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:15.862457 5127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:15.862556 5127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:15.862646 5127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:15.867270 5127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:35963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:15.868693 5127 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:15.869659 5127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:15.876677 5133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:15.876726 5135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:15.876670 5132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:15.877044 5127 server_base.cc:1047] running on GCE node
I20251026 21:06:15.877503 5127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:15.878140 5127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:15.879364 5127 hybrid_clock.cc:648] HybridClock initialized: now 1761512775879352 us; error 33 us; skew 500 ppm
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:15.882516 5127 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:15.883869 5127 webserver.cc:492] Webserver started at http://127.3.175.2:42043/ using document root <none> and password file <none>
I20251026 21:06:15.884456 5127 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:15.884514 5127 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:15.884719 5127 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:15.886672 5127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "eae95a6123c34f3f9c4e0184d39095ea"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "dcd2aa19fb672ec907a614e24369a17c"
server_key_iv: "264dcf2ba0c558611d8c8f4d302a35a4"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.887231 5127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "eae95a6123c34f3f9c4e0184d39095ea"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "dcd2aa19fb672ec907a614e24369a17c"
server_key_iv: "264dcf2ba0c558611d8c8f4d302a35a4"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.891228 5127 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20251026 21:06:15.894021 5142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:15.895239 5127 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251026 21:06:15.895458 5127 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "eae95a6123c34f3f9c4e0184d39095ea"
format_stamp: "Formatted at 2025-10-26 21:06:15 on dist-test-slave-th6x"
server_key: "dcd2aa19fb672ec907a614e24369a17c"
server_key_iv: "264dcf2ba0c558611d8c8f4d302a35a4"
server_key_version: "encryptionkey@0"
I20251026 21:06:15.895572 5127 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:15.912914 5127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:15.916114 5127 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:15.916327 5127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:15.917882 5127 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:15.917941 5127 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:15.918004 5127 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:15.918040 5127 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:15 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:15.929713 5127 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:46593
I20251026 21:06:15.930076 5257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:46593 every 8 connection(s)
I20251026 21:06:15.931043 5127 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:15.933089 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.918696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:53765 (local address 127.3.175.62:35963)
1026 21:06:15.918863 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:15.918868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:15.919769 (+ 901us) server_negotiation.cc:408] Connection header received
1026 21:06:15.920946 (+ 1177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:15.920950 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:15.921022 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:15.921119 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:15.923130 (+ 2011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.923919 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.924913 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.925129 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.928906 (+ 3777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:15.928940 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:15.928944 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:15.928984 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:15.930891 (+ 1907us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.931508 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.931512 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.931514 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.931577 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.932135 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.932138 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.932139 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.932305 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:15.932386 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:15.932850 (+ 464us) server_negotiation.cc:300] Negotiation successful
1026 21:06:15.932964 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251026 21:06:15.933750 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.919051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.2:53765)
1026 21:06:15.919590 (+ 539us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:15.919638 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:15.920683 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:15.921330 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:15.921345 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:15.922041 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:15.922895 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.922916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.924074 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.924079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:15.924785 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.924797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.925082 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.925901 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:15.925981 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:15.928709 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:15.931039 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.931044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.931053 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.931394 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.931701 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.931704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.931706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.931809 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.932401 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:15.932407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:15.932631 (+ 224us) client_negotiation.cc:769] Sending connection context
1026 21:06:15.932807 (+ 176us) client_negotiation.cc:240] Negotiation successful
1026 21:06:15.933033 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20251026 21:06:15.934870 5258 heartbeater.cc:344] Connected to a master server at 127.3.175.62:35963
I20251026 21:06:15.935031 5258 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:15.935587 5258 heartbeater.cc:507] Master 127.3.175.62:35963 requested a full tablet report, sending...
I20251026 21:06:15.936633 4911 ts_manager.cc:194] Registered new tserver with Master: eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593)
I20251026 21:06:15.937383 4911 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:53765
I20251026 21:06:15.938930 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5127
I20251026 21:06:15.939040 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:15.939265 3772 external_mini_cluster.cc:1468] Setting key f6f88033d14d04e32d8c3ec869438b56
I20251026 21:06:15.943962 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.936880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.2:34915)
1026 21:06:15.937148 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:15.937161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:15.937243 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:15.937447 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:15.937450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:15.937656 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:15.937906 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.937913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.938885 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.938889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:15.939455 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:15.939465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.939621 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.940243 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:15.940255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:15.940635 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:15.942572 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.942576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.942579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.942760 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.943020 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:15.943023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:15.943025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:15.943077 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:15.943533 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:15.943535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:15.943613 (+ 78us) client_negotiation.cc:769] Sending connection context
1026 21:06:15.943698 (+ 85us) client_negotiation.cc:240] Negotiation successful
1026 21:06:15.943808 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20251026 21:06:15.944624 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:15.936966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:34915 (local address 127.3.175.62:35963)
1026 21:06:15.937091 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:15.937095 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:15.937219 (+ 124us) server_negotiation.cc:408] Connection header received
1026 21:06:15.937308 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:15.937312 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:15.937356 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:15.937444 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:15.938014 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.938754 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:15.939616 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:15.939838 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:15.940751 (+ 913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:15.940765 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:15.940767 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:15.940790 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:15.942458 (+ 1668us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.942855 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.942861 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.942866 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.942928 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:15.943162 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:15.943168 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:15.943172 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:15.943435 (+ 263us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:15.943550 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:15.943766 (+ 216us) server_negotiation.cc:300] Negotiation successful
1026 21:06:15.943899 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:15.988559 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:35963
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:16.101189 5269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:16.101454 5269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:16.101518 5269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:16.105008 5269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:16.105082 5269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:16.105137 5269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:16.105180 5269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:16.109687 5269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:35963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:16.110855 5269 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:16.111864 5269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:16.118793 5275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:16.119005 5269 server_base.cc:1047] running on GCE node
W20251026 21:06:16.118774 5277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:16.118777 5274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:16.119563 5269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:16.120149 5269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:16.121304 5269 hybrid_clock.cc:648] HybridClock initialized: now 1761512776121293 us; error 27 us; skew 500 ppm
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:16.124109 5269 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:16.125245 5269 webserver.cc:492] Webserver started at http://127.3.175.3:45307/ using document root <none> and password file <none>
I20251026 21:06:16.125821 5269 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:16.125871 5269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:16.126070 5269 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:16.127913 5269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
format_stamp: "Formatted at 2025-10-26 21:06:16 on dist-test-slave-th6x"
server_key: "1379021912530d9131f4dd32560d021f"
server_key_iv: "4d73af04d994e0f586457c2be5718db9"
server_key_version: "encryptionkey@0"
I20251026 21:06:16.128397 5269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
format_stamp: "Formatted at 2025-10-26 21:06:16 on dist-test-slave-th6x"
server_key: "1379021912530d9131f4dd32560d021f"
server_key_iv: "4d73af04d994e0f586457c2be5718db9"
server_key_version: "encryptionkey@0"
I20251026 21:06:16.132056 5269 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:16.134446 5284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:16.135548 5269 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:16.135658 5269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
format_stamp: "Formatted at 2025-10-26 21:06:16 on dist-test-slave-th6x"
server_key: "1379021912530d9131f4dd32560d021f"
server_key_iv: "4d73af04d994e0f586457c2be5718db9"
server_key_version: "encryptionkey@0"
I20251026 21:06:16.135797 5269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:16.159860 5269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:16.167567 5269 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:16.167788 5269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:16.169359 5269 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:16.169413 5269 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:16.169456 5269 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:16.169474 5269 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:16.180860 5269 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:46025
I20251026 21:06:16.180878 5399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:46025 every 8 connection(s)
I20251026 21:06:16.181895 5269 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:16.182585 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.170072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:48425 (local address 127.3.175.62:35963)
1026 21:06:16.170223 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.170228 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.170944 (+ 716us) server_negotiation.cc:408] Connection header received
1026 21:06:16.171898 (+ 954us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.171902 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.171981 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.172089 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.173602 (+ 1513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.174486 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.175362 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.175556 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.178275 (+ 2719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.178297 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.178300 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.178329 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.180385 (+ 2056us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.181040 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.181044 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.181046 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.181100 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.181414 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.181417 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.181419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.181592 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.181697 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.182260 (+ 563us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.182409 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 21:06:16.183324 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.170323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.3:48425)
1026 21:06:16.170771 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.170815 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.171683 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.172291 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.172305 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.172846 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:16.173439 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.173452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.174624 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.174630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.175216 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.175225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.175497 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.176239 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:16.176280 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:16.178092 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:16.180564 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.180571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.180583 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.180894 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.181195 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.181200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.181205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.181316 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.181700 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:16.181710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:16.181973 (+ 263us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.182215 (+ 242us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.182523 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":118,"threads_started":1}
I20251026 21:06:16.184602 5400 heartbeater.cc:344] Connected to a master server at 127.3.175.62:35963
I20251026 21:06:16.184798 5400 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:16.184798 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5269
I20251026 21:06:16.184901 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:16.185175 3772 external_mini_cluster.cc:1468] Setting key 39532833387927bb1bdef7187c272835
I20251026 21:06:16.185467 5400 heartbeater.cc:507] Master 127.3.175.62:35963 requested a full tablet report, sending...
I20251026 21:06:16.186479 4910 ts_manager.cc:194] Registered new tserver with Master: 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.187121 4910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:48425
I20251026 21:06:16.188566 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:06:16.194578 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.186765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.3.175.3:44875)
1026 21:06:16.187606 (+ 841us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.187621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.187749 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.188111 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.188114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.188351 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:16.188620 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.188634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.189403 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.189407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.189803 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.189810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.189931 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.190617 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:16.190637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:16.191122 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:16.193058 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.193063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.193065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.193310 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.193625 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.193629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.193631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.193697 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.194130 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:16.194133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:16.194221 (+ 88us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.194305 (+ 84us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.194424 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":93,"threads_started":1}
I20251026 21:06:16.194765 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.186845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:44875 (local address 127.3.175.62:35963)
1026 21:06:16.186994 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.186997 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.187706 (+ 709us) server_negotiation.cc:408] Connection header received
1026 21:06:16.187863 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.187867 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.187912 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.187972 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.188771 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.189299 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.189910 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.190116 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.191255 (+ 1139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.191272 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.191275 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.191338 (+ 63us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.192938 (+ 1600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.193412 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.193417 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.193421 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.193499 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.193785 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.193791 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.193795 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.194014 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.194127 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.194375 (+ 248us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.194510 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:16.215103 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.207106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.3.175.62:35963)
1026 21:06:16.207244 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.207248 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.207263 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:06:16.207352 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.207356 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.207405 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.207481 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.208423 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.208957 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.209683 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.209816 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.212187 (+ 2371us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.212210 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.212213 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.212246 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.213775 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.214185 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.214187 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.214189 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.214228 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.214385 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.214387 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.214388 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.214536 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.214634 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.214822 (+ 188us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.214925 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:06:16.217980 4910 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:16.220536 4910 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:16.233417 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.228872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:46593 (local address 127.0.0.1:52420)
1026 21:06:16.229411 (+ 539us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.229476 (+ 65us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.229627 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.230545 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.230550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.230570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.230807 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.230813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.232124 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.232127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.232949 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.232958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.233063 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.233135 (+ 72us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.233223 (+ 88us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.233299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":101,"threads_started":1}
I20251026 21:06:16.233807 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.229042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:36411 (local address 127.0.0.1:59140)
1026 21:06:16.230124 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.230138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.230219 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.230879 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.230885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.230907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.231156 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.231162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.232499 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.232502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.233430 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.233439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.233551 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.233569 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.233616 (+ 47us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.233669 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20251026 21:06:16.234557 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.229624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:46025 (local address 127.0.0.1:51228)
1026 21:06:16.230824 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.230839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.230942 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.231380 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.231383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.231403 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.231636 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.231642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.233058 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.233062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.234210 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.234219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.234321 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.234342 (+ 21us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.234392 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.234443 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20251026 21:06:16.234771 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.229558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.3.175.2:46593)
1026 21:06:16.229996 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.230002 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.230029 (+ 27us) server_negotiation.cc:408] Connection header received
1026 21:06:16.230117 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.230123 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.230325 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.230547 (+ 222us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.230966 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.232013 (+ 1047us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.233847 (+ 1834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.234383 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.234465 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.234532 (+ 67us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.234625 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20251026 21:06:16.234972 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.229197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.3.175.1:36411)
1026 21:06:16.230451 (+ 1254us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.230459 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.230482 (+ 23us) server_negotiation.cc:408] Connection header received
1026 21:06:16.230564 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.230569 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.230740 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.230884 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.231334 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.232357 (+ 1023us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.233679 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.234341 (+ 662us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.234465 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.234533 (+ 68us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.234829 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":77,"threads_started":1}
I20251026 21:06:16.236129 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.230043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.3.175.3:46025)
1026 21:06:16.230401 (+ 358us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.230406 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.231035 (+ 629us) server_negotiation.cc:408] Connection header received
1026 21:06:16.231106 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.231109 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.231228 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.231390 (+ 162us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.231759 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.232929 (+ 1170us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.234355 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.235035 (+ 680us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.235833 (+ 798us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.235902 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.235993 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20251026 21:06:16.237829 5192 tablet_service.cc:1505] Processing CreateTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 (DEFAULT_TABLE table=test-table [id=8ae47920b5734904b0fe73efb06dd69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.237880 5050 tablet_service.cc:1505] Processing CreateTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 (DEFAULT_TABLE table=test-table [id=8ae47920b5734904b0fe73efb06dd69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.238687 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012e04ac3abd43c2834e70ec9ab50087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.238775 5050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012e04ac3abd43c2834e70ec9ab50087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.238801 5334 tablet_service.cc:1505] Processing CreateTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 (DEFAULT_TABLE table=test-table [id=8ae47920b5734904b0fe73efb06dd69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.239701 5334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012e04ac3abd43c2834e70ec9ab50087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.244375 5423 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Bootstrap starting.
I20251026 21:06:16.245711 5424 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Bootstrap starting.
I20251026 21:06:16.245918 5425 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Bootstrap starting.
I20251026 21:06:16.247833 5425 tablet_bootstrap.cc:654] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.247964 5424 tablet_bootstrap.cc:654] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.247953 5423 tablet_bootstrap.cc:654] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.249085 5423 log.cc:826] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:16.249085 5424 log.cc:826] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:16.249186 5425 log.cc:826] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:16.251122 5425 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: No bootstrap required, opened a new log
I20251026 21:06:16.251122 5423 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: No bootstrap required, opened a new log
I20251026 21:06:16.251165 5424 tablet_bootstrap.cc:492] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: No bootstrap required, opened a new log
I20251026 21:06:16.251360 5423 ts_tablet_manager.cc:1403] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251026 21:06:16.251360 5425 ts_tablet_manager.cc:1403] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 21:06:16.251379 5424 ts_tablet_manager.cc:1403] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 21:06:16.254004 5424 raft_consensus.cc:359] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.254225 5424 raft_consensus.cc:385] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.254289 5424 raft_consensus.cc:740] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c084fdd4c674b1abe9ed62a06b2a7e9, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.254524 5425 raft_consensus.cc:359] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.254524 5423 raft_consensus.cc:359] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.254736 5423 raft_consensus.cc:385] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.254736 5425 raft_consensus.cc:385] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.254797 5425 raft_consensus.cc:740] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7ad84cd92b4ae2a1f36d605cac23fa, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.254797 5423 raft_consensus.cc:740] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae95a6123c34f3f9c4e0184d39095ea, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.254763 5424 consensus_queue.cc:260] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.255249 5423 consensus_queue.cc:260] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.255249 5425 consensus_queue.cc:260] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.255762 5424 ts_tablet_manager.cc:1434] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251026 21:06:16.255860 5116 heartbeater.cc:499] Master 127.3.175.62:35963 was elected leader, sending a full tablet report...
I20251026 21:06:16.256125 5425 ts_tablet_manager.cc:1434] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:06:16.256493 5400 heartbeater.cc:499] Master 127.3.175.62:35963 was elected leader, sending a full tablet report...
I20251026 21:06:16.256893 5258 heartbeater.cc:499] Master 127.3.175.62:35963 was elected leader, sending a full tablet report...
I20251026 21:06:16.257320 5423 ts_tablet_manager.cc:1434] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 21:06:16.310582 5430 raft_consensus.cc:493] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:16.310799 5430 raft_consensus.cc:515] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.311975 5430 leader_election.cc:290] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1:36411), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.314870 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.312228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:36411 (local address 127.3.175.2:37117)
1026 21:06:16.312393 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.312412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.312526 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.312882 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.312885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.312905 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.313137 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.313143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.313882 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.313888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.314472 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.314479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.314592 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.314617 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.314671 (+ 54us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.314733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20251026 21:06:16.315204 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.312526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:46025 (local address 127.3.175.2:45567)
1026 21:06:16.312928 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.312939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.313033 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.313262 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.313265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.313282 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.313522 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.313528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.314258 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.314261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.314836 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.314843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.314949 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.314968 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.315020 (+ 52us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.315072 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":158,"threads_started":1}
I20251026 21:06:16.315404 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.312339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:37117 (local address 127.3.175.1:36411)
1026 21:06:16.312493 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.312498 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.312514 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:06:16.312695 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.312702 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.312765 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.312863 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.313260 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.313772 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.314596 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.315087 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.315135 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.315197 (+ 62us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.315262 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:06:16.315725 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.312669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:45567 (local address 127.3.175.3:46025)
1026 21:06:16.312837 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.312843 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.313013 (+ 170us) server_negotiation.cc:408] Connection header received
1026 21:06:16.313123 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.313127 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.313173 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.313241 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.313634 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.314141 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.314958 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.315487 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.315529 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.315577 (+ 48us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.315636 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:06:16.316227 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012e04ac3abd43c2834e70ec9ab50087" candidate_uuid: "eae95a6123c34f3f9c4e0184d39095ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" is_pre_election: true
I20251026 21:06:16.316330 5068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012e04ac3abd43c2834e70ec9ab50087" candidate_uuid: "eae95a6123c34f3f9c4e0184d39095ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" is_pre_election: true
I20251026 21:06:16.316506 5354 raft_consensus.cc:2468] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae95a6123c34f3f9c4e0184d39095ea in term 0.
I20251026 21:06:16.316579 5068 raft_consensus.cc:2468] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae95a6123c34f3f9c4e0184d39095ea in term 0.
I20251026 21:06:16.316987 5145 leader_election.cc:304] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7ad84cd92b4ae2a1f36d605cac23fa, eae95a6123c34f3f9c4e0184d39095ea; no voters:
I20251026 21:06:16.317245 5430 raft_consensus.cc:2804] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:16.317327 5430 raft_consensus.cc:493] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:16.317382 5430 raft_consensus.cc:3060] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.318451 5430 raft_consensus.cc:515] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.318794 5430 leader_election.cc:290] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [CANDIDATE]: Term 1 election: Requested vote from peers 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1:36411), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.319192 5068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012e04ac3abd43c2834e70ec9ab50087" candidate_uuid: "eae95a6123c34f3f9c4e0184d39095ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9"
I20251026 21:06:16.319231 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012e04ac3abd43c2834e70ec9ab50087" candidate_uuid: "eae95a6123c34f3f9c4e0184d39095ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
I20251026 21:06:16.319339 5068 raft_consensus.cc:3060] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.319391 5354 raft_consensus.cc:3060] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.320362 5068 raft_consensus.cc:2468] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae95a6123c34f3f9c4e0184d39095ea in term 1.
I20251026 21:06:16.320369 5354 raft_consensus.cc:2468] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae95a6123c34f3f9c4e0184d39095ea in term 1.
I20251026 21:06:16.320691 5145 leader_election.cc:304] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7ad84cd92b4ae2a1f36d605cac23fa, eae95a6123c34f3f9c4e0184d39095ea; no voters:
I20251026 21:06:16.320853 5430 raft_consensus.cc:2804] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:16.321072 5430 raft_consensus.cc:697] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 1 LEADER]: Becoming Leader. State: Replica: eae95a6123c34f3f9c4e0184d39095ea, State: Running, Role: LEADER
I20251026 21:06:16.321378 5430 consensus_queue.cc:237] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.324631 4910 catalog_manager.cc:5649] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea reported cstate change: term changed from 0 to 1, leader changed from <none> to eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2). New cstate: current_term: 1 leader_uuid: "eae95a6123c34f3f9c4e0184d39095ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:16.361851 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.358726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.3.175.2:46593)
1026 21:06:16.358862 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.358865 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.358880 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:06:16.359052 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.359055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.359115 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.359205 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:16.359621 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.360180 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.360795 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.360934 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.360988 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.361489 (+ 501us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.361610 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.361650 (+ 40us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.361737 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251026 21:06:16.367635 5068 raft_consensus.cc:1275] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Refusing update from remote peer eae95a6123c34f3f9c4e0184d39095ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.367636 5354 raft_consensus.cc:1275] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Refusing update from remote peer eae95a6123c34f3f9c4e0184d39095ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.368395 5433 consensus_queue.cc:1048] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.368616 5430 consensus_queue.cc:1048] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.377446 5437 mvcc.cc:204] Tried to move back new op lower bound from 7215156331993874432 to 7215156331815251968. Current Snapshot: MvccSnapshot[applied={T|T < 7215156331993874432}]
I20251026 21:06:16.377661 5438 mvcc.cc:204] Tried to move back new op lower bound from 7215156331993874432 to 7215156331815251968. Current Snapshot: MvccSnapshot[applied={T|T < 7215156331993874432}]
I20251026 21:06:16.380074 5436 mvcc.cc:204] Tried to move back new op lower bound from 7215156331993874432 to 7215156331815251968. Current Snapshot: MvccSnapshot[applied={T|T < 7215156331993874432}]
I20251026 21:06:16.390061 4909 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:16.390241 4909 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:16.393023 4909 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Sending DeleteTablet for 3 replicas of tablet 012e04ac3abd43c2834e70ec9ab50087
I20251026 21:06:16.393923 5050 tablet_service.cc:1552] Processing DeleteTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:59140
I20251026 21:06:16.393916 5192 tablet_service.cc:1552] Processing DeleteTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52420
I20251026 21:06:16.393966 5334 tablet_service.cc:1552] Processing DeleteTablet for tablet 012e04ac3abd43c2834e70ec9ab50087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:51228
I20251026 21:06:16.394510 5448 tablet_replica.cc:333] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: stopping tablet replica
I20251026 21:06:16.394757 5446 tablet_replica.cc:333] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: stopping tablet replica
I20251026 21:06:16.394833 5448 raft_consensus.cc:2243] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:16.394909 4909 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 21:06:16.395020 5446 raft_consensus.cc:2243] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:16.395200 5448 raft_consensus.cc:2272] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 21:06:16.395417 4909 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:16.395483 5447 tablet_replica.cc:333] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: stopping tablet replica
I20251026 21:06:16.395725 5447 raft_consensus.cc:2243] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:16.396067 5447 raft_consensus.cc:2272] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:16.396523 5446 raft_consensus.cc:2272] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:16.397958 5446 ts_tablet_manager.cc:1916] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.398221 5447 ts_tablet_manager.cc:1916] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.400925 5446 ts_tablet_manager.cc:1929] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:16.401129 5446 log.cc:1199] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/012e04ac3abd43c2834e70ec9ab50087
I20251026 21:06:16.401211 5448 ts_tablet_manager.cc:1916] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.401593 5446 ts_tablet_manager.cc:1950] T 012e04ac3abd43c2834e70ec9ab50087 P eae95a6123c34f3f9c4e0184d39095ea: Deleting consensus metadata
I20251026 21:06:16.402772 5050 tablet_service.cc:1505] Processing CreateTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 (DEFAULT_TABLE table=test-table [id=04af88f19362449ba4fc8d4d885c593e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.403064 5050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375a0f4f24bc46caa6dddc18677feaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.403759 5447 ts_tablet_manager.cc:1929] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:16.403802 5448 ts_tablet_manager.cc:1929] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:16.403888 5448 log.cc:1199] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/012e04ac3abd43c2834e70ec9ab50087
I20251026 21:06:16.404049 5447 log.cc:1199] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/012e04ac3abd43c2834e70ec9ab50087
I20251026 21:06:16.404233 5448 ts_tablet_manager.cc:1950] T 012e04ac3abd43c2834e70ec9ab50087 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting consensus metadata
I20251026 21:06:16.404484 5447 ts_tablet_manager.cc:1950] T 012e04ac3abd43c2834e70ec9ab50087 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting consensus metadata
I20251026 21:06:16.405450 5334 tablet_service.cc:1505] Processing CreateTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 (DEFAULT_TABLE table=test-table [id=04af88f19362449ba4fc8d4d885c593e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.405535 4897 catalog_manager.cc:4985] TS 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025): tablet 012e04ac3abd43c2834e70ec9ab50087 (table test-table [id=8ae47920b5734904b0fe73efb06dd69e]) successfully deleted
I20251026 21:06:16.405589 5424 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Bootstrap starting.
I20251026 21:06:16.405735 5334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375a0f4f24bc46caa6dddc18677feaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.406544 5424 tablet_bootstrap.cc:654] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.407629 4897 catalog_manager.cc:4985] TS 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1:36411): tablet 012e04ac3abd43c2834e70ec9ab50087 (table test-table [id=8ae47920b5734904b0fe73efb06dd69e]) successfully deleted
I20251026 21:06:16.408178 5424 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: No bootstrap required, opened a new log
I20251026 21:06:16.408185 5192 tablet_service.cc:1505] Processing CreateTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 (DEFAULT_TABLE table=test-table [id=04af88f19362449ba4fc8d4d885c593e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:16.408270 5424 ts_tablet_manager.cc:1403] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 21:06:16.408452 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 375a0f4f24bc46caa6dddc18677feaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.408730 5424 raft_consensus.cc:359] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.408834 5424 raft_consensus.cc:385] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.408875 5424 raft_consensus.cc:740] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c084fdd4c674b1abe9ed62a06b2a7e9, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.409024 5424 consensus_queue.cc:260] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.409265 5424 ts_tablet_manager.cc:1434] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:16.410269 5425 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Bootstrap starting.
I20251026 21:06:16.410640 5423 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Bootstrap starting.
I20251026 21:06:16.411404 4898 catalog_manager.cc:4985] TS eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593): tablet 012e04ac3abd43c2834e70ec9ab50087 (table test-table [id=8ae47920b5734904b0fe73efb06dd69e]) successfully deleted
I20251026 21:06:16.411747 5425 tablet_bootstrap.cc:654] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.411764 5423 tablet_bootstrap.cc:654] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.413355 5423 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: No bootstrap required, opened a new log
I20251026 21:06:16.413455 5423 ts_tablet_manager.cc:1403] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:16.413995 5423 raft_consensus.cc:359] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.414124 5423 raft_consensus.cc:385] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.414173 5423 raft_consensus.cc:740] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae95a6123c34f3f9c4e0184d39095ea, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.414347 5423 consensus_queue.cc:260] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.414456 5425 tablet_bootstrap.cc:492] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: No bootstrap required, opened a new log
I20251026 21:06:16.414546 5425 ts_tablet_manager.cc:1403] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251026 21:06:16.414633 5423 ts_tablet_manager.cc:1434] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:16.415077 5425 raft_consensus.cc:359] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.415184 5425 raft_consensus.cc:385] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.415227 5425 raft_consensus.cc:740] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7ad84cd92b4ae2a1f36d605cac23fa, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.415377 5425 consensus_queue.cc:260] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.415655 5425 ts_tablet_manager.cc:1434] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20251026 21:06:16.433090 5259 tablet.cc:2378] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:16.433140 5401 tablet.cc:2378] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:16.438628 5117 tablet.cc:2378] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:16.515620 5429 raft_consensus.cc:493] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:16.515821 5429 raft_consensus.cc:515] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.516990 5429 leader_election.cc:290] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.520659 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.517205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:46593 (local address 127.3.175.1:56835)
1026 21:06:16.517557 (+ 352us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.517572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.517719 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.518115 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.518118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.518138 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.518436 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.518445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.519415 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.519418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.520220 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.520227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.520345 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.520370 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.520425 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.520489 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
I20251026 21:06:16.521185 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.517615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:56835 (local address 127.3.175.2:46593)
1026 21:06:16.517767 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.517772 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.517788 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:06:16.517849 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.517852 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.517910 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.517992 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.518603 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.519269 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.520353 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.520889 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.520936 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.520993 (+ 57us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.521058 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251026 21:06:16.521772 5212 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375a0f4f24bc46caa6dddc18677feaa0" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae95a6123c34f3f9c4e0184d39095ea" is_pre_election: true
I20251026 21:06:16.521903 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.517269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:46025 (local address 127.3.175.1:37133)
1026 21:06:16.517713 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.517726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.517856 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.518232 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.518234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.518250 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:16.518912 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.518932 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.520023 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.520027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.520872 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.520882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.521598 (+ 716us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.521617 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.521706 (+ 89us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.521769 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":102,"threads_started":1}
I20251026 21:06:16.521958 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.517451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:37133 (local address 127.3.175.3:46025)
1026 21:06:16.517574 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.517580 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.517792 (+ 212us) server_negotiation.cc:408] Connection header received
1026 21:06:16.518014 (+ 222us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.518018 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.518073 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.518123 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:16.519076 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.519883 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.521024 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.521495 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.521529 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.521764 (+ 235us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.521845 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:06:16.522022 5212 raft_consensus.cc:2468] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 0.
I20251026 21:06:16.522498 5004 leader_election.cc:304] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c084fdd4c674b1abe9ed62a06b2a7e9, eae95a6123c34f3f9c4e0184d39095ea; no voters:
I20251026 21:06:16.522555 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375a0f4f24bc46caa6dddc18677feaa0" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" is_pre_election: true
I20251026 21:06:16.522691 5354 raft_consensus.cc:2468] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 0.
I20251026 21:06:16.522770 5429 raft_consensus.cc:2804] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:16.522840 5429 raft_consensus.cc:493] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:16.522884 5429 raft_consensus.cc:3060] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.523916 5429 raft_consensus.cc:515] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.524291 5429 leader_election.cc:290] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 election: Requested vote from peers eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.524674 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375a0f4f24bc46caa6dddc18677feaa0" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
I20251026 21:06:16.524772 5212 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "375a0f4f24bc46caa6dddc18677feaa0" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae95a6123c34f3f9c4e0184d39095ea"
I20251026 21:06:16.524806 5354 raft_consensus.cc:3060] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.524886 5212 raft_consensus.cc:3060] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.526000 5354 raft_consensus.cc:2468] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 1.
I20251026 21:06:16.526012 5212 raft_consensus.cc:2468] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 1.
I20251026 21:06:16.526350 5003 leader_election.cc:304] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7ad84cd92b4ae2a1f36d605cac23fa, 6c084fdd4c674b1abe9ed62a06b2a7e9; no voters:
I20251026 21:06:16.526542 5429 raft_consensus.cc:2804] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:16.526773 5429 raft_consensus.cc:697] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c084fdd4c674b1abe9ed62a06b2a7e9, State: Running, Role: LEADER
I20251026 21:06:16.527040 5429 consensus_queue.cc:237] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.529680 4909 catalog_manager.cc:5649] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } health_report { overall_health: UNKNOWN } } }
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:16.565501 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.557289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35963 (local address 127.0.0.1:44070)
1026 21:06:16.557572 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.557585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.557726 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.557956 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.557959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.558433 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:16.558628 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.558636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.559400 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.559404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.559872 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.559880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.560000 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.560660 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:16.560680 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:16.562323 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:16.564100 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.564108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.564114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.564333 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.564557 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:16.564560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:16.564561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:16.564602 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:16.564943 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:16.564951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:16.565046 (+ 95us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.565146 (+ 100us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.565309 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
I20251026 21:06:16.565501 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.557335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.3.175.62:35963)
1026 21:06:16.557476 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.557482 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.557658 (+ 176us) server_negotiation.cc:408] Connection header received
1026 21:06:16.557813 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.557816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.557858 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.557942 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.558762 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.559255 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.559975 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.560147 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.562451 (+ 2304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.562472 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.562474 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.562497 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.563987 (+ 1490us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.564424 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.564429 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.564431 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.564468 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.564682 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.564685 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.564687 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.564837 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.564926 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.565196 (+ 270us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.565383 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":45}
I20251026 21:06:16.570086 4909 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:44070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 21:06:16.570896 4909 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:16.575914 5050 tablet_service.cc:1505] Processing CreateTablet for tablet 54378ad8985641c1a706b6d1d2ea5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec947e960f0a44fab552c6e6b2440f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:16.576097 5192 tablet_service.cc:1505] Processing CreateTablet for tablet 54378ad8985641c1a706b6d1d2ea5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec947e960f0a44fab552c6e6b2440f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:16.576325 5050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54378ad8985641c1a706b6d1d2ea5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.576493 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54378ad8985641c1a706b6d1d2ea5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.578573 5423 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: Bootstrap starting.
I20251026 21:06:16.579241 5334 tablet_service.cc:1505] Processing CreateTablet for tablet 54378ad8985641c1a706b6d1d2ea5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec947e960f0a44fab552c6e6b2440f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:16.579519 5423 tablet_bootstrap.cc:654] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.579649 5334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54378ad8985641c1a706b6d1d2ea5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:16.580634 5423 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: No bootstrap required, opened a new log
I20251026 21:06:16.580682 5424 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Bootstrap starting.
I20251026 21:06:16.580730 5423 ts_tablet_manager.cc:1403] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:16.581094 5423 raft_consensus.cc:359] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.581199 5423 raft_consensus.cc:385] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.581234 5423 raft_consensus.cc:740] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae95a6123c34f3f9c4e0184d39095ea, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.581365 5423 consensus_queue.cc:260] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.581615 5423 ts_tablet_manager.cc:1434] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:16.581854 5424 tablet_bootstrap.cc:654] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.581871 5430 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.581979 5430 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.582882 5425 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Bootstrap starting.
I20251026 21:06:16.583258 5424 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: No bootstrap required, opened a new log
I20251026 21:06:16.583410 5424 ts_tablet_manager.cc:1403] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20251026 21:06:16.584025 5425 tablet_bootstrap.cc:654] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:16.583954 5424 raft_consensus.cc:359] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.584087 5424 raft_consensus.cc:385] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.584132 5424 raft_consensus.cc:740] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c084fdd4c674b1abe9ed62a06b2a7e9, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.584262 5424 consensus_queue.cc:260] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.584484 5429 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.584539 5424 ts_tablet_manager.cc:1434] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 21:06:16.584599 5429 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.585760 5425 tablet_bootstrap.cc:492] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: No bootstrap required, opened a new log
I20251026 21:06:16.585855 5425 ts_tablet_manager.cc:1403] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:16.586329 5425 raft_consensus.cc:359] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.586423 5425 raft_consensus.cc:385] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:16.586445 5425 raft_consensus.cc:740] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7ad84cd92b4ae2a1f36d605cac23fa, State: Initialized, Role: FOLLOWER
I20251026 21:06:16.586575 5425 consensus_queue.cc:260] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.586817 5425 ts_tablet_manager.cc:1434] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 21:06:16.586824 5431 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.586999 5431 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.726814 5429 raft_consensus.cc:493] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:16.726964 5429 raft_consensus.cc:515] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.727452 5429 leader_election.cc:290] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.728001 5212 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54378ad8985641c1a706b6d1d2ea5106" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae95a6123c34f3f9c4e0184d39095ea" is_pre_election: true
I20251026 21:06:16.728003 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54378ad8985641c1a706b6d1d2ea5106" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" is_pre_election: true
I20251026 21:06:16.728156 5212 raft_consensus.cc:2468] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 0.
I20251026 21:06:16.728194 5354 raft_consensus.cc:2468] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 0.
I20251026 21:06:16.728492 5004 leader_election.cc:304] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c084fdd4c674b1abe9ed62a06b2a7e9, eae95a6123c34f3f9c4e0184d39095ea; no voters:
I20251026 21:06:16.728677 5429 raft_consensus.cc:2804] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:16.728744 5429 raft_consensus.cc:493] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:16.728772 5429 raft_consensus.cc:3060] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.729652 5429 raft_consensus.cc:515] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.729996 5429 leader_election.cc:290] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 election: Requested vote from peers eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593), 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025)
I20251026 21:06:16.730383 5354 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54378ad8985641c1a706b6d1d2ea5106" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa"
I20251026 21:06:16.730394 5212 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54378ad8985641c1a706b6d1d2ea5106" candidate_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae95a6123c34f3f9c4e0184d39095ea"
I20251026 21:06:16.730494 5212 raft_consensus.cc:3060] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.730504 5354 raft_consensus.cc:3060] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:16.731422 5212 raft_consensus.cc:2468] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 1.
I20251026 21:06:16.731431 5354 raft_consensus.cc:2468] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c084fdd4c674b1abe9ed62a06b2a7e9 in term 1.
I20251026 21:06:16.731735 5004 leader_election.cc:304] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c084fdd4c674b1abe9ed62a06b2a7e9, eae95a6123c34f3f9c4e0184d39095ea; no voters:
I20251026 21:06:16.731899 5429 raft_consensus.cc:2804] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:16.732003 5429 raft_consensus.cc:697] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c084fdd4c674b1abe9ed62a06b2a7e9, State: Running, Role: LEADER
I20251026 21:06:16.732152 5429 consensus_queue.cc:237] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } }
I20251026 21:06:16.732856 5454 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: TxnStatusTablet state changed. Reason: New leader 6c084fdd4c674b1abe9ed62a06b2a7e9. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.732975 5454 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:16.733239 5466 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 21:06:16.733561 5466 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 21:06:16.734035 4909 catalog_manager.cc:5649] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:16.780041 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.776607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.3.175.1:36411)
1026 21:06:16.776745 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.776749 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.776787 (+ 38us) server_negotiation.cc:408] Connection header received
1026 21:06:16.777047 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.777050 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.777112 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.777200 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:16.777607 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.778112 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.778988 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.779113 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.779162 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.779663 (+ 501us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.779761 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.779871 (+ 110us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.779923 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:06:16.780090 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.776454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:36411 (local address 127.0.0.1:59144)
1026 21:06:16.776723 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.776738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.776869 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.777229 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.777233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.777242 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:16.777503 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.777514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.778218 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.778222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.778875 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.778884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.779013 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.779066 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.779766 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.779783 (+ 17us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.779858 (+ 75us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.779948 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20251026 21:06:16.783201 5354 raft_consensus.cc:1275] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Refusing update from remote peer 6c084fdd4c674b1abe9ed62a06b2a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.783391 5212 raft_consensus.cc:1275] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Refusing update from remote peer 6c084fdd4c674b1abe9ed62a06b2a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.783834 5454 consensus_queue.cc:1048] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.784029 5429 consensus_queue.cc:1048] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.785491 5431 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: TxnStatusTablet state changed. Reason: New leader 6c084fdd4c674b1abe9ed62a06b2a7e9. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.785622 5431 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.787369 5430 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: TxnStatusTablet state changed. Reason: New leader 6c084fdd4c674b1abe9ed62a06b2a7e9. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.787469 5430 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.789113 5454 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.789255 5454 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:16.791849 5429 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.791961 5429 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 6c084fdd4c674b1abe9ed62a06b2a7e9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:16.792732 5431 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.792842 5431 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P 5f7ad84cd92b4ae2a1f36d605cac23fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.793578 5430 tablet_replica.cc:442] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c084fdd4c674b1abe9ed62a06b2a7e9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 36411 } } peers { permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 } } peers { permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 } } }
I20251026 21:06:16.793701 5430 tablet_replica.cc:445] T 54378ad8985641c1a706b6d1d2ea5106 P eae95a6123c34f3f9c4e0184d39095ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:16.798916 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.796194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.3.175.1:36411)
1026 21:06:16.796449 (+ 255us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.796453 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.796471 (+ 18us) server_negotiation.cc:408] Connection header received
1026 21:06:16.796506 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.796508 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.796550 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.796613 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:16.797055 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.797510 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.798358 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.798479 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.798516 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.798580 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.798635 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.798741 (+ 106us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.798780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20251026 21:06:16.809325 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.806579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:50001 (local address 127.3.175.1:36411)
1026 21:06:16.806719 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.806724 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.806880 (+ 156us) server_negotiation.cc:408] Connection header received
1026 21:06:16.807030 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.807033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.807074 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.807132 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:16.807547 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.808013 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.808649 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.808836 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.808872 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.808934 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.808999 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.809112 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.809150 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251026 21:06:16.809336 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.806500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:36411 (local address 127.3.175.1:50001)
1026 21:06:16.806797 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:16.806810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:16.806921 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:16.807156 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:16.807159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:16.807167 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:16.807433 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.807440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.808127 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.808129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:16.808543 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:16.808548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.808657 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.808707 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:16.809027 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:16.809034 (+ 7us) client_negotiation.cc:769] Sending connection context
1026 21:06:16.809094 (+ 60us) client_negotiation.cc:240] Negotiation successful
1026 21:06:16.809161 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":131,"threads_started":1}
I20251026 21:06:16.816603 5212 raft_consensus.cc:1275] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Refusing update from remote peer 6c084fdd4c674b1abe9ed62a06b2a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.816617 5354 raft_consensus.cc:1275] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Refusing update from remote peer 6c084fdd4c674b1abe9ed62a06b2a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:16.816999 5429 consensus_queue.cc:1048] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7ad84cd92b4ae2a1f36d605cac23fa" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.817173 5471 consensus_queue.cc:1048] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae95a6123c34f3f9c4e0184d39095ea" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:16.859464 4910 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:16.859651 4910 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:16.861871 4910 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e5c2664154ad43838bb5e38252c2f3e6: Sending DeleteTablet for 3 replicas of tablet 375a0f4f24bc46caa6dddc18677feaa0
I20251026 21:06:16.862550 5192 tablet_service.cc:1552] Processing DeleteTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52420
I20251026 21:06:16.862550 5049 tablet_service.cc:1552] Processing DeleteTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:59140
I20251026 21:06:16.862762 5446 tablet_replica.cc:333] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: stopping tablet replica
I20251026 21:06:16.862779 5448 tablet_replica.cc:333] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: stopping tablet replica
I20251026 21:06:16.862867 5446 raft_consensus.cc:2243] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:16.862875 5448 raft_consensus.cc:2243] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:16.863008 5446 raft_consensus.cc:2272] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:16.863116 5448 raft_consensus.cc:2272] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:16.863615 5334 tablet_service.cc:1552] Processing DeleteTablet for tablet 375a0f4f24bc46caa6dddc18677feaa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:16 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:51228
I20251026 21:06:16.863826 5447 tablet_replica.cc:333] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: stopping tablet replica
I20251026 21:06:16.863925 5447 raft_consensus.cc:2243] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:16.864033 5447 raft_consensus.cc:2272] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:16.864066 5448 ts_tablet_manager.cc:1916] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.864069 5446 ts_tablet_manager.cc:1916] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.864900 5447 ts_tablet_manager.cc:1916] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:16.866724 5448 ts_tablet_manager.cc:1929] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:16.866771 5446 ts_tablet_manager.cc:1929] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:16.866796 5448 log.cc:1199] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/375a0f4f24bc46caa6dddc18677feaa0
I20251026 21:06:16.866842 5446 log.cc:1199] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/375a0f4f24bc46caa6dddc18677feaa0
I20251026 21:06:16.867069 5448 ts_tablet_manager.cc:1950] T 375a0f4f24bc46caa6dddc18677feaa0 P 6c084fdd4c674b1abe9ed62a06b2a7e9: Deleting consensus metadata
I20251026 21:06:16.867106 5446 ts_tablet_manager.cc:1950] T 375a0f4f24bc46caa6dddc18677feaa0 P eae95a6123c34f3f9c4e0184d39095ea: Deleting consensus metadata
I20251026 21:06:16.867709 5447 ts_tablet_manager.cc:1929] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:16.867789 5447 log.cc:1199] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/375a0f4f24bc46caa6dddc18677feaa0
I20251026 21:06:16.867957 4898 catalog_manager.cc:4985] TS eae95a6123c34f3f9c4e0184d39095ea (127.3.175.2:46593): tablet 375a0f4f24bc46caa6dddc18677feaa0 (table test-table [id=04af88f19362449ba4fc8d4d885c593e]) successfully deleted
I20251026 21:06:16.868027 5447 ts_tablet_manager.cc:1950] T 375a0f4f24bc46caa6dddc18677feaa0 P 5f7ad84cd92b4ae2a1f36d605cac23fa: Deleting consensus metadata
I20251026 21:06:16.869128 4897 catalog_manager.cc:4985] TS 5f7ad84cd92b4ae2a1f36d605cac23fa (127.3.175.3:46025): tablet 375a0f4f24bc46caa6dddc18677feaa0 (table test-table [id=04af88f19362449ba4fc8d4d885c593e]) successfully deleted
I20251026 21:06:16.869298 4897 catalog_manager.cc:4985] TS 6c084fdd4c674b1abe9ed62a06b2a7e9 (127.3.175.1:36411): tablet 375a0f4f24bc46caa6dddc18677feaa0 (table test-table [id=04af88f19362449ba4fc8d4d885c593e]) successfully deleted
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:16.880841 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.869312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.3.175.1:36411)
1026 21:06:16.869436 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.869441 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.871671 (+ 2230us) server_negotiation.cc:408] Connection header received
1026 21:06:16.871786 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.871788 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.871839 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.871919 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.873005 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.873468 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.874374 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.874515 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.877313 (+ 2798us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.877343 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.877356 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.877386 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.879219 (+ 1833us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.879725 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.879731 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.879732 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.879788 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.880010 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.880012 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.880013 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.880192 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.880361 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.880538 (+ 177us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.880651 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20251026 21:06:16.881368 5110 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59160
I20251026 21:06:16.890348 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:16.883065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.3.175.62:35963)
1026 21:06:16.883212 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:16.883217 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:16.883232 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:06:16.883413 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:16.883417 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:16.883485 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:16.883576 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:16.884313 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.885049 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:16.885753 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:16.885988 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:16.887009 (+ 1021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:16.887040 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:16.887042 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:16.887072 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:16.888784 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.889161 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.889166 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.889168 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.889220 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:16.889495 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:16.889498 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:16.889500 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:16.889727 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:16.889867 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:16.890057 (+ 190us) server_negotiation.cc:300] Negotiation successful
1026 21:06:16.890183 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20251026 21:06:16.890859 4910 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44082
I20251026 21:06:16.892475 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4985
I20251026 21:06:16.900386 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5127
I20251026 21:06:16.907554 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5269
I20251026 21:06:16.914124 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 4879
2025-10-26T21:06:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3875 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[5484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[5484](info): set up 2 sockets
Oct 26 21:06:16 dist-test-slave-th6x krb5kdc[5484](info): commencing operation
krb5kdc: starting...
W20251026 21:06:18.923982 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:19.068389 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46195
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:19.180954 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:19.181231 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:19.181290 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:19.184782 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:19.184865 5500 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 21:06:19.184885 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:19.184906 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:19.184962 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:19.184996 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:19.189643 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:46195
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:19.190904 5500 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:19.191906 5500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:19.197932 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:19.198046 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.198127 5500 server_base.cc:1047] running on GCE node
W20251026 21:06:19.197934 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.198827 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:19.199875 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:19.201032 5500 hybrid_clock.cc:648] HybridClock initialized: now 1761512779201018 us; error 45 us; skew 500 ppm
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:19.204136 5500 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:19.205341 5500 webserver.cc:492] Webserver started at http://127.3.175.62:43561/ using document root <none> and password file <none>
I20251026 21:06:19.205931 5500 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:19.205996 5500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:19.206195 5500 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:19.208070 5500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "fbabc248863bae47d92f937cc7849612"
server_key_iv: "1b3319d6e1c3ed945f8035d364921cb7"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.208580 5500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "fbabc248863bae47d92f937cc7849612"
server_key_iv: "1b3319d6e1c3ed945f8035d364921cb7"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.212268 5500 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:19.214708 5515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:19.215833 5500 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:19.215988 5500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "fbabc248863bae47d92f937cc7849612"
server_key_iv: "1b3319d6e1c3ed945f8035d364921cb7"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.216101 5500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:19.238813 5500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:19.242161 5500 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:19.242359 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:19.250993 5577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:46195 every 8 connection(s)
I20251026 21:06:19.251001 5500 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:46195
I20251026 21:06:19.252192 5500 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:19.254832 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5500
I20251026 21:06:19.254979 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:19.255244 3772 external_mini_cluster.cc:1468] Setting key d181e862ac11846df305b956edaebc38
I20251026 21:06:19.255820 5578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:19.262580 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Bootstrap starting.
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:19.265231 5578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:19.266078 5578 log.cc:826] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:19.268357 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: No bootstrap required, opened a new log
I20251026 21:06:19.269037 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.256633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.3.175.62:46195)
1026 21:06:19.257052 (+ 419us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:19.257064 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:19.257101 (+ 37us) server_negotiation.cc:408] Connection header received
1026 21:06:19.257793 (+ 692us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:19.257814 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:19.258144 (+ 330us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:19.258442 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:19.259640 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.260418 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.261306 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.261508 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.264251 (+ 2743us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:19.264298 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:19.264310 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:19.264350 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:19.266407 (+ 2057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.266875 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.266882 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.266889 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.266987 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.267236 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.267238 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.267239 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.267639 (+ 400us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:19.267805 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:19.268095 (+ 290us) server_negotiation.cc:300] Negotiation successful
1026 21:06:19.268353 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20251026 21:06:19.271454 5578 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } }
I20251026 21:06:19.271675 5578 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:19.271751 5578 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9cfcfb292a4250a9a5fce4c5cbbc9a, State: Initialized, Role: FOLLOWER
I20251026 21:06:19.272182 5578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } }
I20251026 21:06:19.272318 5578 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:19.272388 5578 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:19.272488 5578 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:19.273536 5578 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } }
I20251026 21:06:19.273905 5578 leader_election.cc:304] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9cfcfb292a4250a9a5fce4c5cbbc9a; no voters:
I20251026 21:06:19.274210 5578 leader_election.cc:290] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:19.274343 5583 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:19.274557 5583 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [term 1 LEADER]: Becoming Leader. State: Replica: 3a9cfcfb292a4250a9a5fce4c5cbbc9a, State: Running, Role: LEADER
I20251026 21:06:19.274865 5583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } }
I20251026 21:06:19.275436 5578 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:19.276491 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9cfcfb292a4250a9a5fce4c5cbbc9a. Latest consensus state: current_term: 1 leader_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } } }
I20251026 21:06:19.276607 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:19.276882 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9cfcfb292a4250a9a5fce4c5cbbc9a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46195 } } }
I20251026 21:06:19.276975 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:19.277060 5592 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:19.280212 5592 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:19.285965 5592 catalog_manager.cc:1357] Generated new cluster ID: b8d1524cac654029b042ca3d43d402f9
I20251026 21:06:19.286063 5592 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:19.292815 5592 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:19.293807 5592 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:19.311753 5592 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Generated new TSK 0
I20251026 21:06:19.312569 5592 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:19.378283 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46195
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:19.506989 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:19.507262 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:19.507371 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:19.510736 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:19.510820 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:19.510893 5606 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 21:06:19.510910 5606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:19.510952 5606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:19.515584 5606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:19.516739 5606 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:19.517624 5606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:19.524674 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:19.524684 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.524900 5606 server_base.cc:1047] running on GCE node
W20251026 21:06:19.524690 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.525393 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:19.526021 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:19.527177 5606 hybrid_clock.cc:648] HybridClock initialized: now 1761512779527169 us; error 27 us; skew 500 ppm
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:19.530130 5606 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:19.531364 5606 webserver.cc:492] Webserver started at http://127.3.175.1:39229/ using document root <none> and password file <none>
I20251026 21:06:19.532011 5606 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:19.532068 5606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:19.532265 5606 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:19.534096 5606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "546e3341248e4386ade22da1745d87f9"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "2eae9e73992573eba2d3f68fb4775a91"
server_key_iv: "91b9a6a4da30bf71f2ec0caaad10b911"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.534588 5606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "546e3341248e4386ade22da1745d87f9"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "2eae9e73992573eba2d3f68fb4775a91"
server_key_iv: "91b9a6a4da30bf71f2ec0caaad10b911"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.538390 5606 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251026 21:06:19.540985 5621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:19.542124 5606 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:19.542249 5606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "546e3341248e4386ade22da1745d87f9"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "2eae9e73992573eba2d3f68fb4775a91"
server_key_iv: "91b9a6a4da30bf71f2ec0caaad10b911"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.542371 5606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:19.566182 5606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:19.569468 5606 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:19.569674 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:19.571240 5606 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:19.571328 5606 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:19.571393 5606 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:19.571425 5606 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:19.583153 5606 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:41291
I20251026 21:06:19.583199 5736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:41291 every 8 connection(s)
I20251026 21:06:19.584408 5606 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:19.584812 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.572054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:54239 (local address 127.3.175.62:46195)
1026 21:06:19.572222 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:19.572227 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:19.572975 (+ 748us) server_negotiation.cc:408] Connection header received
1026 21:06:19.573883 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:19.573887 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:19.573968 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:19.574077 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:19.575981 (+ 1904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.576788 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.577529 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.577753 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.580673 (+ 2920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:19.580728 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:19.580732 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:19.580770 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:19.582535 (+ 1765us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.583161 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.583168 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.583172 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.583236 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.583618 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.583621 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.583622 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.583798 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:19.583884 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:19.584520 (+ 636us) server_negotiation.cc:300] Negotiation successful
1026 21:06:19.584667 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251026 21:06:19.585557 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.572338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.1:54239)
1026 21:06:19.572791 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:19.572836 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:19.573650 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:19.574269 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:19.574284 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:19.574896 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:19.575803 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.575819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.576922 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.576926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:19.577408 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.577416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.577652 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.578409 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:19.578448 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:19.580461 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:19.582684 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.582691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.582703 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.583047 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.583391 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.583395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.583397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.583519 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.583902 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:19.583909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:19.584226 (+ 317us) client_negotiation.cc:769] Sending connection context
1026 21:06:19.584467 (+ 241us) client_negotiation.cc:240] Negotiation successful
1026 21:06:19.584781 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20251026 21:06:19.585775 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5606
I20251026 21:06:19.585886 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:19.586119 3772 external_mini_cluster.cc:1468] Setting key 0484b459b30f59c188f9dca59e5d70bb
I20251026 21:06:19.586906 5737 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46195
I20251026 21:06:19.587078 5737 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:19.587693 5737 heartbeater.cc:507] Master 127.3.175.62:46195 requested a full tablet report, sending...
I20251026 21:06:19.589125 5531 ts_manager.cc:194] Registered new tserver with Master: 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291)
I20251026 21:06:19.590729 5531 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:54239
I20251026 21:06:19.595917 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.589251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:49255 (local address 127.3.175.62:46195)
1026 21:06:19.589439 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:19.589444 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:19.589676 (+ 232us) server_negotiation.cc:408] Connection header received
1026 21:06:19.589792 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:19.589797 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:19.589857 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:19.589949 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:19.590620 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.591121 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.591889 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.592093 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.593041 (+ 948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:19.593058 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:19.593060 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:19.593083 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:19.594483 (+ 1400us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.594870 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.594874 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.594875 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.594920 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.595137 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.595140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.595142 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.595329 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:19.595422 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:19.595658 (+ 236us) server_negotiation.cc:300] Negotiation successful
1026 21:06:19.595779 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251026 21:06:19.595935 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.589156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.1:49255)
1026 21:06:19.589578 (+ 422us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:19.589593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:19.589696 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:19.589984 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:19.589987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:19.590201 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:19.590499 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.590506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.591248 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.591251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:19.591766 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.591775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.591905 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.592544 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:19.592557 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:19.592905 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:19.594604 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.594608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.594610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.594779 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.595007 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.595012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.595014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.595058 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.595416 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:19.595419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:19.595506 (+ 87us) client_negotiation.cc:769] Sending connection context
1026 21:06:19.595603 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 21:06:19.595742 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:19.637974 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46195
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:19.750882 5748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:19.751140 5748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:19.751209 5748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:19.754842 5748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:19.754926 5748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:19.754984 5748 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 21:06:19.755000 5748 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:19.755041 5748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:19.759545 5748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:19.760676 5748 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:19.761513 5748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:19.768388 5754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:19.768337 5753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.768797 5748 server_base.cc:1047] running on GCE node
W20251026 21:06:19.768980 5756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:19.769352 5748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:19.769907 5748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:19.771066 5748 hybrid_clock.cc:648] HybridClock initialized: now 1761512779771053 us; error 28 us; skew 500 ppm
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:19.773768 5748 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:19.774813 5748 webserver.cc:492] Webserver started at http://127.3.175.2:46259/ using document root <none> and password file <none>
I20251026 21:06:19.775389 5748 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:19.775442 5748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:19.775624 5748 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:19.777495 5748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "8a4161c48ab347f280ae2cfc032ba19e"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "39e36accc2f241f2ba73a35f896976a6"
server_key_iv: "9a40e63f3a5298bf51b6dc568516ae24"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.777920 5748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "8a4161c48ab347f280ae2cfc032ba19e"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "39e36accc2f241f2ba73a35f896976a6"
server_key_iv: "9a40e63f3a5298bf51b6dc568516ae24"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.781364 5748 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:19.783703 5763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:19.784740 5748 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:19.784854 5748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "8a4161c48ab347f280ae2cfc032ba19e"
format_stamp: "Formatted at 2025-10-26 21:06:19 on dist-test-slave-th6x"
server_key: "39e36accc2f241f2ba73a35f896976a6"
server_key_iv: "9a40e63f3a5298bf51b6dc568516ae24"
server_key_version: "encryptionkey@0"
I20251026 21:06:19.784955 5748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:19.800134 5748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:19.803254 5748 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:19.803509 5748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:19.805045 5748 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:19.805101 5748 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:19.805163 5748 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:19.805195 5748 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:19 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:19.815531 5748 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:41147
I20251026 21:06:19.815594 5878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:41147 every 8 connection(s)
I20251026 21:06:19.816506 5748 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:19.819330 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.805928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:53661 (local address 127.3.175.62:46195)
1026 21:06:19.806083 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:19.806091 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:19.806945 (+ 854us) server_negotiation.cc:408] Connection header received
1026 21:06:19.807958 (+ 1013us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:19.807962 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:19.808032 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:19.808113 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:19.810103 (+ 1990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.810975 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.812129 (+ 1154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.812370 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.815290 (+ 2920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:19.815344 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:19.815348 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:19.815382 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:19.817081 (+ 1699us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.817581 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.817585 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.817587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.817639 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.818005 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.818009 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.818011 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.818250 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:19.818356 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:19.819001 (+ 645us) server_negotiation.cc:300] Negotiation successful
1026 21:06:19.819136 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:06:19.820017 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.806264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.2:53661)
1026 21:06:19.806794 (+ 530us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:19.806832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:19.807720 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:19.808409 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:19.808424 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:19.809064 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:19.809884 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.809914 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.811146 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.811150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:19.811836 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.811849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.812210 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.813089 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:19.813115 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:19.815078 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:19.817235 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.817239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.817248 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.817481 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.817723 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.817728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.817749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.817881 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.818377 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:19.818384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:19.818703 (+ 319us) client_negotiation.cc:769] Sending connection context
1026 21:06:19.818949 (+ 246us) client_negotiation.cc:240] Negotiation successful
1026 21:06:19.819226 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20251026 21:06:19.821130 5879 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46195
I20251026 21:06:19.821310 5879 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:19.821846 5879 heartbeater.cc:507] Master 127.3.175.62:46195 requested a full tablet report, sending...
I20251026 21:06:19.822876 5532 ts_manager.cc:194] Registered new tserver with Master: 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147)
I20251026 21:06:19.823609 5532 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:53661
I20251026 21:06:19.824326 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5748
I20251026 21:06:19.824429 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:19.824746 3772 external_mini_cluster.cc:1468] Setting key 13c940e6e8d86bd890598975a3435c8c
I20251026 21:06:19.830156 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.823228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.2:46907)
1026 21:06:19.823584 (+ 356us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:19.823597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:19.823703 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:19.823983 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:19.823987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:19.824217 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:19.824451 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.824460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.825210 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.825213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:19.825675 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:19.825684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.825801 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.826364 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:19.826377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:19.826679 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:19.828854 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.828858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.828859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.829036 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.829281 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:19.829283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:19.829285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:19.829322 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:19.829736 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:19.829738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:19.829811 (+ 73us) client_negotiation.cc:769] Sending connection context
1026 21:06:19.829886 (+ 75us) client_negotiation.cc:240] Negotiation successful
1026 21:06:19.830005 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20251026 21:06:19.830317 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:19.823326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:46907 (local address 127.3.175.62:46195)
1026 21:06:19.823458 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:19.823476 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:19.823670 (+ 194us) server_negotiation.cc:408] Connection header received
1026 21:06:19.823797 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:19.823803 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:19.823858 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:19.823947 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:19.824573 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.825103 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:19.825902 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:19.826080 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:19.826811 (+ 731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:19.826830 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:19.826837 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:19.826868 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:19.828744 (+ 1876us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.829130 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.829137 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.829142 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.829197 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:19.829390 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:19.829396 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:19.829401 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:19.829652 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:19.829759 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:19.829963 (+ 204us) server_negotiation.cc:300] Negotiation successful
1026 21:06:19.830106 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:19.873844 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46195
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:19.987918 5890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:19.988178 5890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:19.988250 5890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:19.991798 5890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:19.991883 5890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:19.991940 5890 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 21:06:19.991962 5890 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:06:19.992009 5890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:19.996423 5890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:19.997532 5890 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:19.998409 5890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:20.005295 5895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:20.005296 5896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:20.005296 5898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:20.005367 5890 server_base.cc:1047] running on GCE node
I20251026 21:06:20.005965 5890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:20.006541 5890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:20.007716 5890 hybrid_clock.cc:648] HybridClock initialized: now 1761512780007708 us; error 41 us; skew 500 ppm
Oct 26 21:06:20 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512780, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:20.010471 5890 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:20.011638 5890 webserver.cc:492] Webserver started at http://127.3.175.3:37793/ using document root <none> and password file <none>
I20251026 21:06:20.012203 5890 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:20.012260 5890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:20.012449 5890 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:20.014242 5890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
format_stamp: "Formatted at 2025-10-26 21:06:20 on dist-test-slave-th6x"
server_key: "02832b74f0ae72a4292e38ada4cfef01"
server_key_iv: "93d9b080ed75cbd2d7da6c7d3a99e5d8"
server_key_version: "encryptionkey@0"
I20251026 21:06:20.014734 5890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
format_stamp: "Formatted at 2025-10-26 21:06:20 on dist-test-slave-th6x"
server_key: "02832b74f0ae72a4292e38ada4cfef01"
server_key_iv: "93d9b080ed75cbd2d7da6c7d3a99e5d8"
server_key_version: "encryptionkey@0"
I20251026 21:06:20.018260 5890 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:20.020593 5905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:20.022022 5890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:20.022153 5890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
format_stamp: "Formatted at 2025-10-26 21:06:20 on dist-test-slave-th6x"
server_key: "02832b74f0ae72a4292e38ada4cfef01"
server_key_iv: "93d9b080ed75cbd2d7da6c7d3a99e5d8"
server_key_version: "encryptionkey@0"
I20251026 21:06:20.022295 5890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:20.036007 5890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:20.039117 5890 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:20.039354 5890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:20.041051 5890 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:20.041110 5890 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:20.041169 5890 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:20.041203 5890 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:06:20 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512780, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:20.052524 5890 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:41427
I20251026 21:06:20.052769 6020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:41427 every 8 connection(s)
I20251026 21:06:20.053529 5890 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:20.054013 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.042906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:37985 (local address 127.3.175.62:46195)
1026 21:06:20.043086 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.043092 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.043114 (+ 22us) server_negotiation.cc:408] Connection header received
1026 21:06:20.043578 (+ 464us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.043586 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.043653 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.043757 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:20.045253 (+ 1496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.046167 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.047142 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.047414 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.049745 (+ 2331us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:20.049773 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:20.049779 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:20.049818 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:20.051941 (+ 2123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.052522 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.052528 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.052530 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.052577 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.052911 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.052914 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.052915 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.053092 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:20.053188 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.053689 (+ 501us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.053825 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251026 21:06:20.054651 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.041929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.3:37985)
1026 21:06:20.042403 (+ 474us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.042449 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.043337 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.043964 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.043976 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.044498 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:20.045083 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.045097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.046350 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.046355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.047004 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.047019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.047316 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.048033 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:20.048068 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:20.049551 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:20.052102 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.052109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.052121 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.052422 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.052667 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.052670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.052673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.052794 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.053202 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:20.053210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:20.053450 (+ 240us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.053639 (+ 189us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.053934 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20251026 21:06:20.055851 6021 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46195
I20251026 21:06:20.056018 6021 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:20.056526 6021 heartbeater.cc:507] Master 127.3.175.62:46195 requested a full tablet report, sending...
I20251026 21:06:20.057636 5531 ts_manager.cc:194] Registered new tserver with Master: ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:20.058326 5531 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:37985
I20251026 21:06:20.059989 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 5890
I20251026 21:06:20.060108 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:20.060406 3772 external_mini_cluster.cc:1468] Setting key 28a9015eda84588e030412878ee5c52b
I20251026 21:06:20.062718 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:06:20.065670 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.057880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:60955 (local address 127.3.175.62:46195)
1026 21:06:20.058016 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.058020 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.058100 (+ 80us) server_negotiation.cc:408] Connection header received
1026 21:06:20.058257 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.058260 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.058304 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.058372 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:20.059004 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.059539 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.060333 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.060527 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.061931 (+ 1404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:20.061945 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:20.061948 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:20.061977 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:20.064078 (+ 2101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.064469 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.064475 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.064479 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.064533 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.064746 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.064752 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.064757 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.065029 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:20.065148 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.065417 (+ 269us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.065520 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 21:06:20.065696 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.057790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.3.175.3:60955)
1026 21:06:20.058024 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.058040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.058165 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.058396 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.058399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.058572 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:20.058848 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.058856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.059639 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.059643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.060189 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.060201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.060653 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.061347 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:20.061374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:20.061826 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:20.064197 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.064200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.064202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.064376 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.064617 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.064619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.064620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.064659 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.065152 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:20.065159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:20.065262 (+ 103us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.065359 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.065512 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
Oct 26 21:06:20 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 21:06:20 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:20.087426 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.080341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.3.175.62:46195)
1026 21:06:20.080470 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.080473 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.080488 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:06:20.080542 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.080545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.080593 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.080674 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:20.081441 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.081945 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.082506 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.082627 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.084843 (+ 2216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:20.084864 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:20.084867 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:20.084893 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:20.086204 (+ 1311us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.086554 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.086556 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.086558 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.086597 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.086758 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.086760 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.086761 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.086902 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:20.086999 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.087153 (+ 154us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.087247 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:06:20.090148 5531 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:20.092646 5531 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:20.105749 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.101068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41147 (local address 127.0.0.1:50524)
1026 21:06:20.101704 (+ 636us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.101748 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.101901 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.102796 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.102800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.102819 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:20.103024 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.103030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104573 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.104576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.105308 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.105317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.105411 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.105429 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.105477 (+ 48us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.105563 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":97,"threads_started":1}
I20251026 21:06:20.106412 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.100923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:41427 (local address 127.0.0.1:52768)
1026 21:06:20.101707 (+ 784us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.101748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.101921 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.102658 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.102666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.102687 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:20.102996 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.103006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104465 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.104469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.105278 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.105286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.106136 (+ 850us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.106153 (+ 17us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.106234 (+ 81us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.106294 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":86,"threads_started":1}
I20251026 21:06:20.106508 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.101091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.3.175.3:41427)
1026 21:06:20.101516 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.101529 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.101832 (+ 303us) server_negotiation.cc:408] Connection header received
1026 21:06:20.102111 (+ 279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.102117 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.102314 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.102462 (+ 148us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:20.103130 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104355 (+ 1225us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.105428 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.105994 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.106123 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.106272 (+ 149us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.106362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":109,"threads_started":1}
I20251026 21:06:20.106545 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.101820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.3.175.2:41147)
1026 21:06:20.102363 (+ 543us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.102370 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.102391 (+ 21us) server_negotiation.cc:408] Connection header received
1026 21:06:20.102470 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.102475 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.102638 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.103170 (+ 532us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:20.103364 (+ 194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104452 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.105651 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.106141 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.106248 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.106307 (+ 59us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.106392 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":121,"threads_started":1}
I20251026 21:06:20.107076 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.101428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.0.0.1:46828)
1026 21:06:20.102309 (+ 881us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.102323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.102413 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.103049 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.103054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.103076 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:20.103246 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.103251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104282 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.104285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.105177 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.105186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.105293 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.105363 (+ 70us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.105462 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.105558 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":101,"threads_started":1}
I20251026 21:06:20.107101 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.101808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.3.175.1:41291)
1026 21:06:20.102161 (+ 353us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.102168 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.102504 (+ 336us) server_negotiation.cc:408] Connection header received
1026 21:06:20.102595 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.102603 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.102783 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.102915 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:20.103427 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.104157 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.106139 (+ 1982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.106748 (+ 609us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.106818 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.106869 (+ 51us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.106951 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":95,"threads_started":1}
I20251026 21:06:20.109542 5955 tablet_service.cc:1505] Processing CreateTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 (DEFAULT_TABLE table=test-table [id=c2c6102b16a940769bc68d880c498cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.109563 5813 tablet_service.cc:1505] Processing CreateTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 (DEFAULT_TABLE table=test-table [id=c2c6102b16a940769bc68d880c498cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.109642 5671 tablet_service.cc:1505] Processing CreateTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 (DEFAULT_TABLE table=test-table [id=c2c6102b16a940769bc68d880c498cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.110405 5955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267e62ccacf46c58c1285ce8eb15686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.110436 5671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267e62ccacf46c58c1285ce8eb15686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.110436 5813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267e62ccacf46c58c1285ce8eb15686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.116199 6044 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Bootstrap starting.
I20251026 21:06:20.116811 6046 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Bootstrap starting.
I20251026 21:06:20.116978 6045 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Bootstrap starting.
I20251026 21:06:20.118886 6045 tablet_bootstrap.cc:654] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.118973 6046 tablet_bootstrap.cc:654] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.119509 6044 tablet_bootstrap.cc:654] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.119690 6045 log.cc:826] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:20.120298 6046 log.cc:826] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:20.120710 6044 log.cc:826] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:20.121491 6045 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: No bootstrap required, opened a new log
I20251026 21:06:20.121692 6045 ts_tablet_manager.cc:1403] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 21:06:20.122089 6046 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: No bootstrap required, opened a new log
I20251026 21:06:20.122263 6046 ts_tablet_manager.cc:1403] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251026 21:06:20.122430 6044 tablet_bootstrap.cc:492] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: No bootstrap required, opened a new log
I20251026 21:06:20.122596 6044 ts_tablet_manager.cc:1403] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251026 21:06:20.124682 6045 raft_consensus.cc:359] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.124866 6045 raft_consensus.cc:385] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.124925 6045 raft_consensus.cc:740] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.125337 6045 consensus_queue.cc:260] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.125442 6044 raft_consensus.cc:359] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.125442 6046 raft_consensus.cc:359] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.125643 6046 raft_consensus.cc:385] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.125643 6044 raft_consensus.cc:385] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.125705 6046 raft_consensus.cc:740] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4161c48ab347f280ae2cfc032ba19e, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.125717 6044 raft_consensus.cc:740] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.126124 5737 heartbeater.cc:499] Master 127.3.175.62:46195 was elected leader, sending a full tablet report...
I20251026 21:06:20.126150 6045 ts_tablet_manager.cc:1434] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251026 21:06:20.126145 6044 consensus_queue.cc:260] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.126602 6046 consensus_queue.cc:260] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.126829 6021 heartbeater.cc:499] Master 127.3.175.62:46195 was elected leader, sending a full tablet report...
I20251026 21:06:20.127054 6044 ts_tablet_manager.cc:1434] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20251026 21:06:20.127329 5879 heartbeater.cc:499] Master 127.3.175.62:46195 was elected leader, sending a full tablet report...
I20251026 21:06:20.127441 6046 ts_tablet_manager.cc:1434] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20251026 21:06:20.305069 6022 tablet.cc:2378] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:20.314860 6050 raft_consensus.cc:493] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:20.315071 6050 raft_consensus.cc:515] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.316273 6050 leader_election.cc:290] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427), 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147)
W20251026 21:06:20.317700 5880 tablet.cc:2378] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:20.319854 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.316560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41147 (local address 127.3.175.1:49171)
1026 21:06:20.316951 (+ 391us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.316964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.317108 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.317357 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.317360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.317386 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:20.317681 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.317692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.318711 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.318715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.319533 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.319541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.319635 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.319653 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.319688 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.319736 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":12672,"thread_start_us":125,"threads_started":1}
I20251026 21:06:20.320475 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.316529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:41427 (local address 127.3.175.1:53701)
1026 21:06:20.316938 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.316953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.317105 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.317357 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.317360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.317389 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:20.318208 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.318216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.319167 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.319171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.320009 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.320021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.320174 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.320195 (+ 21us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.320251 (+ 56us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.320312 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"spinlock_wait_cycles":28928,"thread_start_us":201,"threads_started":1}
I20251026 21:06:20.320642 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.316661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:49171 (local address 127.3.175.2:41147)
1026 21:06:20.316813 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.316817 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.317053 (+ 236us) server_negotiation.cc:408] Connection header received
1026 21:06:20.317222 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.317225 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.317270 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.317338 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:20.317839 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.318562 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.319736 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.320323 (+ 587us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.320363 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.320417 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.320496 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251026 21:06:20.320936 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.316621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:53701 (local address 127.3.175.3:41427)
1026 21:06:20.316769 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.316774 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.317046 (+ 272us) server_negotiation.cc:408] Connection header received
1026 21:06:20.317217 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.317225 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.317270 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.317341 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:20.318347 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.319022 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.320166 (+ 1144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.320679 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.320731 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.320785 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.320848 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:06:20.321296 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0267e62ccacf46c58c1285ce8eb15686" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e" is_pre_election: true
I20251026 21:06:20.321588 5833 raft_consensus.cc:2468] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.321575 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0267e62ccacf46c58c1285ce8eb15686" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" is_pre_election: true
I20251026 21:06:20.321813 5975 raft_consensus.cc:2468] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.322276 5623 leader_election.cc:304] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:20.322551 6050 raft_consensus.cc:2804] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:20.322638 6050 raft_consensus.cc:493] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:20.322686 6050 raft_consensus.cc:3060] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.323668 6050 raft_consensus.cc:515] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.324028 6050 leader_election.cc:290] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Requested vote from peers ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427), 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147)
I20251026 21:06:20.324357 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0267e62ccacf46c58c1285ce8eb15686" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
I20251026 21:06:20.324469 5975 raft_consensus.cc:3060] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.324533 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0267e62ccacf46c58c1285ce8eb15686" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e"
I20251026 21:06:20.324654 5833 raft_consensus.cc:3060] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.325732 5975 raft_consensus.cc:2468] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.325747 5833 raft_consensus.cc:2468] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.326067 5622 leader_election.cc:304] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e; no voters:
I20251026 21:06:20.326319 6050 raft_consensus.cc:2804] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:20.326529 6050 raft_consensus.cc:697] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Becoming Leader. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Running, Role: LEADER
I20251026 21:06:20.326848 6050 consensus_queue.cc:237] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } }
I20251026 21:06:20.330178 5531 catalog_manager.cc:5649] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546e3341248e4386ade22da1745d87f9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } health_report { overall_health: HEALTHY } } }
W20251026 21:06:20.336702 5738 tablet.cc:2378] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 26 21:06:20 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:20.373344 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.365246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46195 (local address 127.0.0.1:40260)
1026 21:06:20.365536 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.365549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.365696 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.365948 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.365951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.366381 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:20.366573 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.366579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.367323 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.367329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.367743 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.367752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.367849 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.368493 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:20.368507 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:20.369960 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:20.371979 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.371986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.371991 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.372203 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.372435 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:20.372438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:20.372439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:20.372484 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:20.372845 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:20.372851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:20.372951 (+ 100us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.373041 (+ 90us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.373164 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20251026 21:06:20.373353 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.365346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.3.175.62:46195)
1026 21:06:20.365492 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.365497 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.365623 (+ 126us) server_negotiation.cc:408] Connection header received
1026 21:06:20.365792 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.365795 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.365837 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.365911 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:20.366703 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.367183 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.367856 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.368010 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.370085 (+ 2075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:20.370105 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:20.370107 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:20.370132 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:20.371869 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.372299 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.372302 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.372303 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.372350 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:20.372552 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:20.372554 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:20.372555 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:20.372754 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:20.372840 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.373112 (+ 272us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.373225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20251026 21:06:20.377671 5531 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:40260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 21:06:20.378458 5531 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:20.383816 5671 tablet_service.cc:1505] Processing CreateTablet for tablet a8781a4483bc48f98d01f275d99065dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08930cf2cdf94da08e29df4441895362]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:20.383997 5813 tablet_service.cc:1505] Processing CreateTablet for tablet a8781a4483bc48f98d01f275d99065dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08930cf2cdf94da08e29df4441895362]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:20.384177 5955 tablet_service.cc:1505] Processing CreateTablet for tablet a8781a4483bc48f98d01f275d99065dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08930cf2cdf94da08e29df4441895362]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:06:20.384498 5671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8781a4483bc48f98d01f275d99065dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.384634 5955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8781a4483bc48f98d01f275d99065dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.384683 5813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8781a4483bc48f98d01f275d99065dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.386864 6045 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: Bootstrap starting.
I20251026 21:06:20.387354 6046 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: Bootstrap starting.
I20251026 21:06:20.388037 6045 tablet_bootstrap.cc:654] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.388447 6046 tablet_bootstrap.cc:654] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.388825 6044 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: Bootstrap starting.
I20251026 21:06:20.389551 6045 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: No bootstrap required, opened a new log
I20251026 21:06:20.389631 6045 ts_tablet_manager.cc:1403] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:20.389720 6046 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: No bootstrap required, opened a new log
I20251026 21:06:20.389791 6046 ts_tablet_manager.cc:1403] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251026 21:06:20.389923 6044 tablet_bootstrap.cc:654] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.390202 6045 raft_consensus.cc:359] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.390318 6045 raft_consensus.cc:385] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.390352 6045 raft_consensus.cc:740] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.390312 6046 raft_consensus.cc:359] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.390416 6046 raft_consensus.cc:385] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.390455 6046 raft_consensus.cc:740] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4161c48ab347f280ae2cfc032ba19e, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.390573 6045 consensus_queue.cc:260] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.390643 6046 consensus_queue.cc:260] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.390923 6045 ts_tablet_manager.cc:1434] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:20.390903 6052 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.390890 6050 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.391007 6052 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.391007 6050 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.391323 6044 tablet_bootstrap.cc:492] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: No bootstrap required, opened a new log
I20251026 21:06:20.391412 6044 ts_tablet_manager.cc:1403] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:20.391799 6044 raft_consensus.cc:359] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.391901 6044 raft_consensus.cc:385] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.391938 6044 raft_consensus.cc:740] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.392092 6044 consensus_queue.cc:260] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.392339 6044 ts_tablet_manager.cc:1434] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:20.392632 6051 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.392760 6051 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.393697 6046 ts_tablet_manager.cc:1434] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 21:06:20.631225 6050 raft_consensus.cc:493] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:20.631446 6050 raft_consensus.cc:515] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.631934 6050 leader_election.cc:290] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:20.632412 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8781a4483bc48f98d01f275d99065dc" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" is_pre_election: true
I20251026 21:06:20.632416 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8781a4483bc48f98d01f275d99065dc" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e" is_pre_election: true
I20251026 21:06:20.632550 5975 raft_consensus.cc:2468] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.632568 5833 raft_consensus.cc:2468] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.632882 5623 leader_election.cc:304] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:20.633071 6050 raft_consensus.cc:2804] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:20.633147 6050 raft_consensus.cc:493] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:20.633180 6050 raft_consensus.cc:3060] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.634173 6050 raft_consensus.cc:515] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.634575 6050 leader_election.cc:290] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:20.634990 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8781a4483bc48f98d01f275d99065dc" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
I20251026 21:06:20.634990 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8781a4483bc48f98d01f275d99065dc" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e"
I20251026 21:06:20.635129 5833 raft_consensus.cc:3060] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.635129 5975 raft_consensus.cc:3060] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.636358 5833 raft_consensus.cc:2468] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.636358 5975 raft_consensus.cc:2468] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.636775 5623 leader_election.cc:304] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:20.636991 6050 raft_consensus.cc:2804] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:20.637086 6050 raft_consensus.cc:697] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Becoming Leader. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Running, Role: LEADER
I20251026 21:06:20.637236 6050 consensus_queue.cc:237] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.637913 6055 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: TxnStatusTablet state changed. Reason: New leader 546e3341248e4386ade22da1745d87f9. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.638048 6055 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:20.638319 6067 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 21:06:20.638581 6067 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 21:06:20.639077 5532 catalog_manager.cc:5649] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546e3341248e4386ade22da1745d87f9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:20.690059 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.686304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.3.175.1:41291)
1026 21:06:20.686654 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.686658 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.686669 (+ 11us) server_negotiation.cc:408] Connection header received
1026 21:06:20.686746 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.686749 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.686813 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.686911 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:20.687273 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.688177 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.688824 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.689059 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.689126 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:20.689604 (+ 478us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:20.689726 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.689837 (+ 111us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.689889 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
I20251026 21:06:20.690044 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.686178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.0.0.1:46836)
1026 21:06:20.686334 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.686351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.686467 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.686936 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.686939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.686955 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:20.687159 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.687165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.688296 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.688299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.688700 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.688707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.688842 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.688916 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:20.689710 (+ 794us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:20.689751 (+ 41us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.689818 (+ 67us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.689905 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251026 21:06:20.695173 5975 raft_consensus.cc:1275] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.695199 5833 raft_consensus.cc:1275] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.695883 6055 consensus_queue.cc:1048] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:20.696089 6050 consensus_queue.cc:1048] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:20.700248 6052 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: TxnStatusTablet state changed. Reason: New leader 546e3341248e4386ade22da1745d87f9. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.700377 6052 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.700683 6051 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: TxnStatusTablet state changed. Reason: New leader 546e3341248e4386ade22da1745d87f9. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.700778 6051 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.704164 6071 mvcc.cc:204] Tried to move back new op lower bound from 7215156349720330240 to 7215156349491650560. Current Snapshot: MvccSnapshot[applied={T|T < 7215156349720330240}]
I20251026 21:06:20.704288 6050 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.704394 6050 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:20.705924 6055 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.706033 6055 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 546e3341248e4386ade22da1745d87f9: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:06:20.706483 6051 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.706607 6051 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P ea99d00a02e14e5f88b87dd52416bdcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.706593 6052 tablet_replica.cc:442] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } } }
I20251026 21:06:20.706686 6052 tablet_replica.cc:445] T a8781a4483bc48f98d01f275d99065dc P 8a4161c48ab347f280ae2cfc032ba19e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:06:20.716476 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.713212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.3.175.1:41291)
1026 21:06:20.713363 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.713370 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.713483 (+ 113us) server_negotiation.cc:408] Connection header received
1026 21:06:20.713612 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.713614 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.713668 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.713743 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:20.714333 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.714849 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.715826 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.716006 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.716053 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:20.716123 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:20.716183 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.716294 (+ 111us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.716335 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251026 21:06:20.727340 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.724160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:35211 (local address 127.3.175.1:41291)
1026 21:06:20.724309 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:20.724314 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:20.724557 (+ 243us) server_negotiation.cc:408] Connection header received
1026 21:06:20.724677 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:20.724679 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:20.724723 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:20.724784 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:20.725285 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.725758 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.726674 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.726814 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.726876 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:20.726960 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:20.727027 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:20.727141 (+ 114us) server_negotiation.cc:300] Negotiation successful
1026 21:06:20.727188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:06:20.727401 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:20.724046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.3.175.1:35211)
1026 21:06:20.724469 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:20.724483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:20.724595 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:20.724806 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:20.724809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:20.724830 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:20.725179 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.725187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.725848 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:20.725851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:20.726564 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:20.726574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:20.726694 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:20.726742 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:20.727042 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:20.727047 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 21:06:20.727112 (+ 65us) client_negotiation.cc:240] Negotiation successful
1026 21:06:20.727183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":125,"threads_started":1}
I20251026 21:06:20.734719 5833 raft_consensus.cc:1275] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.734812 5975 raft_consensus.cc:1275] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.735219 6050 consensus_queue.cc:1048] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:20.735460 6055 consensus_queue.cc:1048] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:20.776340 5532 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:20.776548 5532 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:20.779250 5532 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Sending DeleteTablet for 3 replicas of tablet 0267e62ccacf46c58c1285ce8eb15686
I20251026 21:06:20.780124 5955 tablet_service.cc:1552] Processing DeleteTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:20 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52768
I20251026 21:06:20.780182 5671 tablet_service.cc:1552] Processing DeleteTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:20 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46828
I20251026 21:06:20.780206 5813 tablet_service.cc:1552] Processing DeleteTablet for tablet 0267e62ccacf46c58c1285ce8eb15686 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:20 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:50524
I20251026 21:06:20.780730 6088 tablet_replica.cc:333] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: stopping tablet replica
I20251026 21:06:20.780850 6090 tablet_replica.cc:333] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: stopping tablet replica
I20251026 21:06:20.781127 6088 raft_consensus.cc:2243] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:20.781127 6090 raft_consensus.cc:2243] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:20.781366 6090 raft_consensus.cc:2272] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:20.781366 6088 raft_consensus.cc:2272] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:20.781751 5530 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:20.782274 5530 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:20.782853 6090 ts_tablet_manager.cc:1916] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:20.783375 6088 ts_tablet_manager.cc:1916] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:20.786007 6090 ts_tablet_manager.cc:1929] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:20.786126 6090 log.cc:1199] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/0267e62ccacf46c58c1285ce8eb15686
I20251026 21:06:20.786456 6089 tablet_replica.cc:333] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: stopping tablet replica
I20251026 21:06:20.786767 6090 ts_tablet_manager.cc:1950] T 0267e62ccacf46c58c1285ce8eb15686 P 8a4161c48ab347f280ae2cfc032ba19e: Deleting consensus metadata
I20251026 21:06:20.786934 6089 raft_consensus.cc:2243] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:20.787425 6089 raft_consensus.cc:2272] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:20.788141 5516 catalog_manager.cc:4985] TS 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147): tablet 0267e62ccacf46c58c1285ce8eb15686 (table test-table [id=c2c6102b16a940769bc68d880c498cc4]) successfully deleted
I20251026 21:06:20.788612 6088 ts_tablet_manager.cc:1929] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:20.789096 5813 tablet_service.cc:1505] Processing CreateTablet for tablet f15ef192936c4673b6440e2e60590eaa (DEFAULT_TABLE table=test-table [id=fb0624f6980e433d97cd4daa4505bda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.789340 6089 ts_tablet_manager.cc:1916] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:20.789402 5813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15ef192936c4673b6440e2e60590eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.791785 6046 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Bootstrap starting.
I20251026 21:06:20.792240 6089 ts_tablet_manager.cc:1929] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:20.792351 6089 log.cc:1199] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/0267e62ccacf46c58c1285ce8eb15686
I20251026 21:06:20.792686 6089 ts_tablet_manager.cc:1950] T 0267e62ccacf46c58c1285ce8eb15686 P 546e3341248e4386ade22da1745d87f9: Deleting consensus metadata
I20251026 21:06:20.792953 6046 tablet_bootstrap.cc:654] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.793237 5671 tablet_service.cc:1505] Processing CreateTablet for tablet f15ef192936c4673b6440e2e60590eaa (DEFAULT_TABLE table=test-table [id=fb0624f6980e433d97cd4daa4505bda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.793493 5671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15ef192936c4673b6440e2e60590eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.788920 5955 tablet_service.cc:1505] Processing CreateTablet for tablet f15ef192936c4673b6440e2e60590eaa (DEFAULT_TABLE table=test-table [id=fb0624f6980e433d97cd4daa4505bda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:20.794096 5955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15ef192936c4673b6440e2e60590eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:20.794764 6046 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: No bootstrap required, opened a new log
I20251026 21:06:20.794854 6046 ts_tablet_manager.cc:1403] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251026 21:06:20.795413 6046 raft_consensus.cc:359] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.795526 6046 raft_consensus.cc:385] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.795562 6046 raft_consensus.cc:740] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4161c48ab347f280ae2cfc032ba19e, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.795702 6046 consensus_queue.cc:260] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.795894 6045 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Bootstrap starting.
I20251026 21:06:20.795953 6046 ts_tablet_manager.cc:1434] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:20.796482 6044 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Bootstrap starting.
I20251026 21:06:20.796758 6088 log.cc:1199] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/0267e62ccacf46c58c1285ce8eb15686
I20251026 21:06:20.796948 6045 tablet_bootstrap.cc:654] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.797165 6088 ts_tablet_manager.cc:1950] T 0267e62ccacf46c58c1285ce8eb15686 P ea99d00a02e14e5f88b87dd52416bdcc: Deleting consensus metadata
I20251026 21:06:20.797598 6044 tablet_bootstrap.cc:654] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:20.798743 5517 catalog_manager.cc:4985] TS ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427): tablet 0267e62ccacf46c58c1285ce8eb15686 (table test-table [id=c2c6102b16a940769bc68d880c498cc4]) successfully deleted
I20251026 21:06:20.799016 6044 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: No bootstrap required, opened a new log
I20251026 21:06:20.799121 6044 ts_tablet_manager.cc:1403] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:20.799652 6044 raft_consensus.cc:359] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.799767 6044 raft_consensus.cc:385] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.799790 6044 raft_consensus.cc:740] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.799918 6044 consensus_queue.cc:260] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.800037 5519 catalog_manager.cc:4985] TS 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291): tablet 0267e62ccacf46c58c1285ce8eb15686 (table test-table [id=c2c6102b16a940769bc68d880c498cc4]) successfully deleted
I20251026 21:06:20.800200 6044 ts_tablet_manager.cc:1434] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:20.802453 6045 tablet_bootstrap.cc:492] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: No bootstrap required, opened a new log
I20251026 21:06:20.802570 6045 ts_tablet_manager.cc:1403] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251026 21:06:20.803138 6045 raft_consensus.cc:359] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.803275 6045 raft_consensus.cc:385] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:20.803347 6045 raft_consensus.cc:740] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Initialized, Role: FOLLOWER
I20251026 21:06:20.803514 6045 consensus_queue.cc:260] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.803821 6045 ts_tablet_manager.cc:1434] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:20.931270 6086 raft_consensus.cc:493] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:20.931455 6086 raft_consensus.cc:515] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.931913 6086 leader_election.cc:290] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:20.932420 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f15ef192936c4673b6440e2e60590eaa" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" is_pre_election: true
I20251026 21:06:20.932413 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f15ef192936c4673b6440e2e60590eaa" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e" is_pre_election: true
I20251026 21:06:20.932567 5833 raft_consensus.cc:2468] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.932567 5975 raft_consensus.cc:2468] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:20.932931 5622 leader_election.cc:304] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:20.933131 6086 raft_consensus.cc:2804] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:20.933187 6086 raft_consensus.cc:493] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:20.933238 6086 raft_consensus.cc:3060] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.934171 6086 raft_consensus.cc:515] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.934531 6086 leader_election.cc:290] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:20.934938 5833 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f15ef192936c4673b6440e2e60590eaa" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e"
I20251026 21:06:20.934954 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f15ef192936c4673b6440e2e60590eaa" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
I20251026 21:06:20.935068 5833 raft_consensus.cc:3060] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.935089 5975 raft_consensus.cc:3060] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:20.936018 5833 raft_consensus.cc:2468] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.936246 5975 raft_consensus.cc:2468] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:20.936365 5622 leader_election.cc:304] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e; no voters:
I20251026 21:06:20.936564 6086 raft_consensus.cc:2804] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:20.936653 6086 raft_consensus.cc:697] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Becoming Leader. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Running, Role: LEADER
I20251026 21:06:20.936805 6086 consensus_queue.cc:237] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:20.938664 5532 catalog_manager.cc:5649] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546e3341248e4386ade22da1745d87f9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:20.972733 5975 raft_consensus.cc:1275] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.972752 5833 raft_consensus.cc:1275] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:20.973167 6055 consensus_queue.cc:1048] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:20.973562 6055 consensus_queue.cc:1048] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 21:06:20.991326 6071 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251026 21:06:22.554823 5532 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:22.554996 5532 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:22.557384 5532 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Sending DeleteTablet for 3 replicas of tablet f15ef192936c4673b6440e2e60590eaa
I20251026 21:06:22.558109 5955 tablet_service.cc:1552] Processing DeleteTablet for tablet f15ef192936c4673b6440e2e60590eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52768
I20251026 21:06:22.558102 5813 tablet_service.cc:1552] Processing DeleteTablet for tablet f15ef192936c4673b6440e2e60590eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:50524
I20251026 21:06:22.558110 5670 tablet_service.cc:1552] Processing DeleteTablet for tablet f15ef192936c4673b6440e2e60590eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46828
I20251026 21:06:22.558512 6117 tablet_replica.cc:333] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: stopping tablet replica
I20251026 21:06:22.558749 6116 tablet_replica.cc:333] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: stopping tablet replica
I20251026 21:06:22.558840 6117 raft_consensus.cc:2243] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:22.558908 6116 raft_consensus.cc:2243] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:22.559008 6117 raft_consensus.cc:2272] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.559049 6116 raft_consensus.cc:2272] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.559326 6118 tablet_replica.cc:333] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: stopping tablet replica
I20251026 21:06:22.559479 6118 raft_consensus.cc:2243] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:22.559657 6117 ts_tablet_manager.cc:1916] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.559757 6118 raft_consensus.cc:2272] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.560436 6118 ts_tablet_manager.cc:1916] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.561738 6117 ts_tablet_manager.cc:1929] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 21:06:22.561812 6117 log.cc:1199] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/f15ef192936c4673b6440e2e60590eaa
I20251026 21:06:22.562047 6117 ts_tablet_manager.cc:1950] T f15ef192936c4673b6440e2e60590eaa P ea99d00a02e14e5f88b87dd52416bdcc: Deleting consensus metadata
I20251026 21:06:22.562770 5517 catalog_manager.cc:4985] TS ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427): tablet f15ef192936c4673b6440e2e60590eaa (table test-table [id=fb0624f6980e433d97cd4daa4505bda0]) successfully deleted
I20251026 21:06:22.562971 6118 ts_tablet_manager.cc:1929] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 21:06:22.563043 6118 log.cc:1199] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/f15ef192936c4673b6440e2e60590eaa
I20251026 21:06:22.563350 6118 ts_tablet_manager.cc:1950] T f15ef192936c4673b6440e2e60590eaa P 546e3341248e4386ade22da1745d87f9: Deleting consensus metadata
I20251026 21:06:22.563822 6116 ts_tablet_manager.cc:1916] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.564131 5519 catalog_manager.cc:4985] TS 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291): tablet f15ef192936c4673b6440e2e60590eaa (table test-table [id=fb0624f6980e433d97cd4daa4505bda0]) successfully deleted
I20251026 21:06:22.566053 6116 ts_tablet_manager.cc:1929] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 21:06:22.566123 6116 log.cc:1199] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/f15ef192936c4673b6440e2e60590eaa
I20251026 21:06:22.566319 6116 ts_tablet_manager.cc:1950] T f15ef192936c4673b6440e2e60590eaa P 8a4161c48ab347f280ae2cfc032ba19e: Deleting consensus metadata
I20251026 21:06:22.566970 5516 catalog_manager.cc:4985] TS 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147): tablet f15ef192936c4673b6440e2e60590eaa (table test-table [id=fb0624f6980e433d97cd4daa4505bda0]) successfully deleted
Oct 26 21:06:22 dist-test-slave-th6x krb5kdc[5484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 21:06:22 dist-test-slave-th6x krb5kdc[5484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:22.593282 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.581142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.3.175.62:46195)
1026 21:06:22.581446 (+ 304us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.581451 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.581466 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:06:22.581579 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.581583 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.581634 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.581722 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:22.582563 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.583142 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.583845 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.584070 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.590061 (+ 5991us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:22.590087 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:22.590090 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:22.590124 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:22.591814 (+ 1690us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:22.592274 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:22.592276 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:22.592278 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:22.592322 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:22.592535 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:22.592537 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:22.592538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:22.592691 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:22.592804 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.593015 (+ 211us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.593128 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20251026 21:06:22.595947 5532 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:22.596414 5532 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:22.601200 5670 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf (DEFAULT_TABLE table=test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.601387 5813 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf (DEFAULT_TABLE table=test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.601568 5670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab8ca4b376648cdbf254f8ce533ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.601670 5813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab8ca4b376648cdbf254f8ce533ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.603602 5955 tablet_service.cc:1505] Processing CreateTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf (DEFAULT_TABLE table=test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.603884 5955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab8ca4b376648cdbf254f8ce533ddbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.603906 6127 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Bootstrap starting.
I20251026 21:06:22.604851 6128 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Bootstrap starting.
I20251026 21:06:22.604918 6127 tablet_bootstrap.cc:654] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.605602 6128 tablet_bootstrap.cc:654] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.606320 6127 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: No bootstrap required, opened a new log
I20251026 21:06:22.606324 6130 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Bootstrap starting.
I20251026 21:06:22.606487 6127 ts_tablet_manager.cc:1403] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:22.606868 6128 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: No bootstrap required, opened a new log
I20251026 21:06:22.606974 6128 ts_tablet_manager.cc:1403] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:22.607101 6127 raft_consensus.cc:359] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.607309 6130 tablet_bootstrap.cc:654] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.607391 6127 raft_consensus.cc:385] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.607492 6127 raft_consensus.cc:740] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.607496 6128 raft_consensus.cc:359] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.607612 6128 raft_consensus.cc:385] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.607661 6128 raft_consensus.cc:740] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4161c48ab347f280ae2cfc032ba19e, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.607805 6128 consensus_queue.cc:260] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.607894 6127 consensus_queue.cc:260] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.608258 6128 ts_tablet_manager.cc:1434] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 21:06:22.608537 6130 tablet_bootstrap.cc:492] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: No bootstrap required, opened a new log
I20251026 21:06:22.608618 6130 ts_tablet_manager.cc:1403] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:22.608707 6127 ts_tablet_manager.cc:1434] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:22.609095 6130 raft_consensus.cc:359] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.609196 6130 raft_consensus.cc:385] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.609227 6130 raft_consensus.cc:740] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.609364 6130 consensus_queue.cc:260] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.609697 6130 ts_tablet_manager.cc:1434] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:22.627954 6099 raft_consensus.cc:493] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:22.628108 6099 raft_consensus.cc:515] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.628530 6099 leader_election.cc:290] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:22.629026 5832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab8ca4b376648cdbf254f8ce533ddbf" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e" is_pre_election: true
I20251026 21:06:22.629019 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab8ca4b376648cdbf254f8ce533ddbf" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" is_pre_election: true
I20251026 21:06:22.629168 5832 raft_consensus.cc:2468] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:22.629158 5975 raft_consensus.cc:2468] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 0.
I20251026 21:06:22.629515 5622 leader_election.cc:304] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e; no voters:
I20251026 21:06:22.629709 6099 raft_consensus.cc:2804] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:22.629771 6099 raft_consensus.cc:493] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:22.629799 6099 raft_consensus.cc:3060] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.630756 6099 raft_consensus.cc:515] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.631125 6099 leader_election.cc:290] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147), ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427)
I20251026 21:06:22.631520 5975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab8ca4b376648cdbf254f8ce533ddbf" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99d00a02e14e5f88b87dd52416bdcc"
I20251026 21:06:22.631539 5832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ab8ca4b376648cdbf254f8ce533ddbf" candidate_uuid: "546e3341248e4386ade22da1745d87f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e"
I20251026 21:06:22.631639 5832 raft_consensus.cc:3060] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.631639 5975 raft_consensus.cc:3060] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.632515 5975 raft_consensus.cc:2468] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:22.632516 5832 raft_consensus.cc:2468] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546e3341248e4386ade22da1745d87f9 in term 1.
I20251026 21:06:22.632822 5622 leader_election.cc:304] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e; no voters:
I20251026 21:06:22.632987 6099 raft_consensus.cc:2804] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:22.633093 6099 raft_consensus.cc:697] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Becoming Leader. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Running, Role: LEADER
I20251026 21:06:22.633241 6099 consensus_queue.cc:237] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.634984 5532 catalog_manager.cc:5649] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546e3341248e4386ade22da1745d87f9 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "546e3341248e4386ade22da1745d87f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:22.654383 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.651860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.3.175.1:41291)
1026 21:06:22.652120 (+ 260us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.652123 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.652140 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:06:22.652178 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.652180 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.652226 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.652296 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:22.652605 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.653132 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.653736 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.653873 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.653910 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.653996 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.654053 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.654157 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.654201 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20251026 21:06:22.660918 5975 raft_consensus.cc:1275] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:22.660917 5832 raft_consensus.cc:1275] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Refusing update from remote peer 546e3341248e4386ade22da1745d87f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:22.661295 6099 consensus_queue.cc:1048] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:22.661432 6104 consensus_queue.cc:1048] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:22.664443 6101 mvcc.cc:204] Tried to move back new op lower bound from 7215156357776121856 to 7215156357667495936. Current Snapshot: MvccSnapshot[applied={T|T < 7215156357776121856}]
I20251026 21:06:22.664499 6100 mvcc.cc:204] Tried to move back new op lower bound from 7215156357776121856 to 7215156357667495936. Current Snapshot: MvccSnapshot[applied={T|T < 7215156357776121856}]
I20251026 21:06:22.692509 5532 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:22.692662 5532 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:22.694562 5532 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Sending DeleteTablet for 3 replicas of tablet 4ab8ca4b376648cdbf254f8ce533ddbf
I20251026 21:06:22.695104 5670 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46828
I20251026 21:06:22.695142 5813 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:50524
I20251026 21:06:22.695331 6118 tablet_replica.cc:333] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: stopping tablet replica
I20251026 21:06:22.695274 5955 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ab8ca4b376648cdbf254f8ce533ddbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:22 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52768
I20251026 21:06:22.695438 6116 tablet_replica.cc:333] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: stopping tablet replica
I20251026 21:06:22.695447 6118 raft_consensus.cc:2243] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:22.695482 6117 tablet_replica.cc:333] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: stopping tablet replica
I20251026 21:06:22.695537 6116 raft_consensus.cc:2243] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:22.695601 6117 raft_consensus.cc:2243] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:22.695640 6116 raft_consensus.cc:2272] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.695632 6118 raft_consensus.cc:2272] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.695734 6117 raft_consensus.cc:2272] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:22.695804 5532 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:22.696254 5532 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:22.696508 6118 ts_tablet_manager.cc:1916] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.696518 6117 ts_tablet_manager.cc:1916] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.698769 6117 ts_tablet_manager.cc:1929] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:22.698848 6117 log.cc:1199] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/4ab8ca4b376648cdbf254f8ce533ddbf
I20251026 21:06:22.698972 6118 ts_tablet_manager.cc:1929] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:22.699048 6118 log.cc:1199] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/4ab8ca4b376648cdbf254f8ce533ddbf
I20251026 21:06:22.699120 6117 ts_tablet_manager.cc:1950] T 4ab8ca4b376648cdbf254f8ce533ddbf P ea99d00a02e14e5f88b87dd52416bdcc: Deleting consensus metadata
I20251026 21:06:22.699334 6118 ts_tablet_manager.cc:1950] T 4ab8ca4b376648cdbf254f8ce533ddbf P 546e3341248e4386ade22da1745d87f9: Deleting consensus metadata
I20251026 21:06:22.699888 5517 catalog_manager.cc:4985] TS ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427): tablet 4ab8ca4b376648cdbf254f8ce533ddbf (table test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]) successfully deleted
I20251026 21:06:22.700099 5519 catalog_manager.cc:4985] TS 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291): tablet 4ab8ca4b376648cdbf254f8ce533ddbf (table test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]) successfully deleted
I20251026 21:06:22.700184 6116 ts_tablet_manager.cc:1916] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:22.701421 5670 tablet_service.cc:1505] Processing CreateTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb (DEFAULT_TABLE table=test-table [id=d60fe7b19d62492783042299e20118dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.701685 5670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfaf7af09d474c8c836cf0dbcda876eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.701889 5813 tablet_service.cc:1505] Processing CreateTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb (DEFAULT_TABLE table=test-table [id=d60fe7b19d62492783042299e20118dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.702140 5813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfaf7af09d474c8c836cf0dbcda876eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.702409 6116 ts_tablet_manager.cc:1929] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:06:22.702461 6116 log.cc:1199] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/4ab8ca4b376648cdbf254f8ce533ddbf
I20251026 21:06:22.702637 6116 ts_tablet_manager.cc:1950] T 4ab8ca4b376648cdbf254f8ce533ddbf P 8a4161c48ab347f280ae2cfc032ba19e: Deleting consensus metadata
I20251026 21:06:22.703563 5955 tablet_service.cc:1505] Processing CreateTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb (DEFAULT_TABLE table=test-table [id=d60fe7b19d62492783042299e20118dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:22.703774 5955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfaf7af09d474c8c836cf0dbcda876eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:22.704149 6127 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: Bootstrap starting.
I20251026 21:06:22.704202 6128 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Bootstrap starting.
I20251026 21:06:22.704967 6128 tablet_bootstrap.cc:654] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.705053 6127 tablet_bootstrap.cc:654] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.705821 6130 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Bootstrap starting.
I20251026 21:06:22.705858 5516 catalog_manager.cc:4985] TS 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147): tablet 4ab8ca4b376648cdbf254f8ce533ddbf (table test-table [id=e5cdc90e59e14cf1b7f56afb81f0f66f]) successfully deleted
I20251026 21:06:22.706485 6127 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: No bootstrap required, opened a new log
I20251026 21:06:22.706573 6127 ts_tablet_manager.cc:1403] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:22.706655 6128 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: No bootstrap required, opened a new log
I20251026 21:06:22.706760 6128 ts_tablet_manager.cc:1403] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 21:06:22.706871 6130 tablet_bootstrap.cc:654] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:22.707044 6127 raft_consensus.cc:359] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.707145 6127 raft_consensus.cc:385] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.707185 6127 raft_consensus.cc:740] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546e3341248e4386ade22da1745d87f9, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.707356 6127 consensus_queue.cc:260] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.707334 6128 raft_consensus.cc:359] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.707453 6128 raft_consensus.cc:385] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.707486 6128 raft_consensus.cc:740] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4161c48ab347f280ae2cfc032ba19e, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.707638 6127 ts_tablet_manager.cc:1434] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 21:06:22.707617 6128 consensus_queue.cc:260] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.707877 6128 ts_tablet_manager.cc:1434] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:22.708499 6130 tablet_bootstrap.cc:492] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: No bootstrap required, opened a new log
I20251026 21:06:22.708591 6130 ts_tablet_manager.cc:1403] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:06:22.709056 6130 raft_consensus.cc:359] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.709190 6130 raft_consensus.cc:385] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:22.709225 6130 raft_consensus.cc:740] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Initialized, Role: FOLLOWER
I20251026 21:06:22.709347 6130 consensus_queue.cc:260] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.709640 6130 ts_tablet_manager.cc:1434] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:06:22.851351 6134 raft_consensus.cc:493] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:22.851505 6134 raft_consensus.cc:515] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.852669 6134 leader_election.cc:290] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291), 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147)
I20251026 21:06:22.855890 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.852983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.3.175.3:37673)
1026 21:06:22.853347 (+ 364us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:22.853360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:22.853500 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:22.853771 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:22.853774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:22.853803 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:22.854051 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.854057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.854759 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.854763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:22.855379 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.855388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.855517 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.855541 (+ 24us) client_negotiation.cc:769] Sending connection context
1026 21:06:22.855596 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 21:06:22.855656 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"spinlock_wait_cycles":19584,"thread_start_us":108,"threads_started":1}
I20251026 21:06:22.855912 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.852984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41147 (local address 127.3.175.3:40887)
1026 21:06:22.853345 (+ 361us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:22.853360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:22.853499 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:22.853773 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:22.853776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:22.853805 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:22.854050 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.854056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.854802 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.854805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:22.855408 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.855416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.855522 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.855541 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:06:22.855594 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 21:06:22.855655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":29,"spinlock_wait_cycles":44672,"thread_start_us":106,"threads_started":1}
I20251026 21:06:22.856310 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.853064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:37673 (local address 127.3.175.1:41291)
1026 21:06:22.853224 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.853231 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.853453 (+ 222us) server_negotiation.cc:408] Connection header received
1026 21:06:22.853620 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.853622 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.853671 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.853752 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:22.854173 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.854649 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.855495 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.856002 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.856053 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.856113 (+ 60us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.856188 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251026 21:06:22.856330 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.853055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:40887 (local address 127.3.175.2:41147)
1026 21:06:22.853398 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.853404 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.853421 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:06:22.853621 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.853624 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.853667 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.853744 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:22.854170 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.854683 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.855521 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.856055 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.856090 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.856142 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.856194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20251026 21:06:22.856907 5690 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfaf7af09d474c8c836cf0dbcda876eb" candidate_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546e3341248e4386ade22da1745d87f9" is_pre_election: true
I20251026 21:06:22.856922 5832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfaf7af09d474c8c836cf0dbcda876eb" candidate_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e" is_pre_election: true
I20251026 21:06:22.857072 5832 raft_consensus.cc:2468] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea99d00a02e14e5f88b87dd52416bdcc in term 0.
I20251026 21:06:22.857115 5690 raft_consensus.cc:2468] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea99d00a02e14e5f88b87dd52416bdcc in term 0.
I20251026 21:06:22.857494 5906 leader_election.cc:304] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 546e3341248e4386ade22da1745d87f9, 8a4161c48ab347f280ae2cfc032ba19e, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:22.857748 6134 raft_consensus.cc:2804] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:22.857805 6134 raft_consensus.cc:493] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:22.857834 6134 raft_consensus.cc:3060] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.858775 6134 raft_consensus.cc:515] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.859163 6134 leader_election.cc:290] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [CANDIDATE]: Term 1 election: Requested vote from peers 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291), 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147)
I20251026 21:06:22.859539 5690 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfaf7af09d474c8c836cf0dbcda876eb" candidate_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546e3341248e4386ade22da1745d87f9"
I20251026 21:06:22.859539 5832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfaf7af09d474c8c836cf0dbcda876eb" candidate_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4161c48ab347f280ae2cfc032ba19e"
I20251026 21:06:22.859668 5832 raft_consensus.cc:3060] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.859668 5690 raft_consensus.cc:3060] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:22.860728 5832 raft_consensus.cc:2468] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea99d00a02e14e5f88b87dd52416bdcc in term 1.
I20251026 21:06:22.860792 5690 raft_consensus.cc:2468] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea99d00a02e14e5f88b87dd52416bdcc in term 1.
I20251026 21:06:22.861040 5906 leader_election.cc:304] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4161c48ab347f280ae2cfc032ba19e, ea99d00a02e14e5f88b87dd52416bdcc; no voters:
I20251026 21:06:22.861197 6134 raft_consensus.cc:2804] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:22.861394 6134 raft_consensus.cc:697] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 1 LEADER]: Becoming Leader. State: Replica: ea99d00a02e14e5f88b87dd52416bdcc, State: Running, Role: LEADER
I20251026 21:06:22.861614 6134 consensus_queue.cc:237] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } }
I20251026 21:06:22.863775 5529 catalog_manager.cc:5649] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3). New cstate: current_term: 1 leader_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea99d00a02e14e5f88b87dd52416bdcc" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 41427 } health_report { overall_health: HEALTHY } } }
I20251026 21:06:22.908731 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.905594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.3.175.3:41427)
1026 21:06:22.905897 (+ 303us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.905901 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.905913 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:06:22.906004 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.906006 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.906060 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.906152 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:22.906457 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.907014 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.907618 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.907759 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.907812 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.908318 (+ 506us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.908409 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.908485 (+ 76us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.908533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":137,"threads_started":1}
I20251026 21:06:22.918726 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.915951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:52491 (local address 127.3.175.1:41291)
1026 21:06:22.916112 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.916118 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.916198 (+ 80us) server_negotiation.cc:408] Connection header received
1026 21:06:22.916353 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.916357 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.916404 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.916471 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:22.916868 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.917359 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.918030 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.918230 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.918273 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.918346 (+ 73us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.918415 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.918529 (+ 114us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.918579 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251026 21:06:22.918748 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.915729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.3.175.3:52491)
1026 21:06:22.916097 (+ 368us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:22.916114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:22.916244 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:22.916492 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:22.916495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:22.916503 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:22.916756 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.916761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.917474 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.917477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:22.917913 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.917919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.918028 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.918078 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.918441 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.918448 (+ 7us) client_negotiation.cc:769] Sending connection context
1026 21:06:22.918510 (+ 62us) client_negotiation.cc:240] Negotiation successful
1026 21:06:22.918586 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20251026 21:06:22.925093 5832 raft_consensus.cc:1275] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Refusing update from remote peer ea99d00a02e14e5f88b87dd52416bdcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:22.925388 5690 raft_consensus.cc:1275] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Refusing update from remote peer ea99d00a02e14e5f88b87dd52416bdcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:22.925572 6134 consensus_queue.cc:1048] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4161c48ab347f280ae2cfc032ba19e" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:22.925853 6147 consensus_queue.cc:1048] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "546e3341248e4386ade22da1745d87f9" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 41291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:22.943672 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.940773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:41427 (local address 127.3.175.1:42153)
1026 21:06:22.941045 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:22.941063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:22.941159 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:22.941378 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:22.941380 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:22.941386 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:06:22.941590 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.941594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.942324 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.942327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:22.942798 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:22.942809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.942920 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.942947 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.943359 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.943365 (+ 6us) client_negotiation.cc:769] Sending connection context
1026 21:06:22.943437 (+ 72us) client_negotiation.cc:240] Negotiation successful
1026 21:06:22.943518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20251026 21:06:22.944023 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:22.941112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:42153 (local address 127.3.175.3:41427)
1026 21:06:22.941217 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:22.941220 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:22.941231 (+ 11us) server_negotiation.cc:408] Connection header received
1026 21:06:22.941267 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:22.941269 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:22.941314 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:22.941373 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:22.941670 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.942221 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:22.942913 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:22.943111 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:22.943155 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:22.943226 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:22.943342 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:22.943873 (+ 531us) server_negotiation.cc:300] Negotiation successful
1026 21:06:22.943928 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20251026 21:06:22.944613 6100 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251026 21:06:24.503391 5532 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:24.503556 5532 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40264:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:24.505985 5532 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a9cfcfb292a4250a9a5fce4c5cbbc9a: Sending DeleteTablet for 3 replicas of tablet cfaf7af09d474c8c836cf0dbcda876eb
I20251026 21:06:24.506604 5813 tablet_service.cc:1552] Processing DeleteTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:24 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:50524
I20251026 21:06:24.506604 5670 tablet_service.cc:1552] Processing DeleteTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:24 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46828
I20251026 21:06:24.506707 5955 tablet_service.cc:1552] Processing DeleteTablet for tablet cfaf7af09d474c8c836cf0dbcda876eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:24 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52768
I20251026 21:06:24.507010 6180 tablet_replica.cc:333] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9: stopping tablet replica
I20251026 21:06:24.507071 6182 tablet_replica.cc:333] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: stopping tablet replica
I20251026 21:06:24.507143 6180 raft_consensus.cc:2243] T cfaf7af09d474c8c836cf0dbcda876eb P 546e3341248e4386ade22da1745d87f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:24.507227 6182 raft_consensus.cc:2243] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:24.507369 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5606
I20251026 21:06:24.508783 6181 tablet_replica.cc:333] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: stopping tablet replica
I20251026 21:06:24.508893 6181 raft_consensus.cc:2243] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:24.509006 6181 raft_consensus.cc:2272] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:24.509593 6181 ts_tablet_manager.cc:1916] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:24.511683 6181 ts_tablet_manager.cc:1929] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 21:06:24.511758 6181 log.cc:1199] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/cfaf7af09d474c8c836cf0dbcda876eb
I20251026 21:06:24.512089 6181 ts_tablet_manager.cc:1950] T cfaf7af09d474c8c836cf0dbcda876eb P 8a4161c48ab347f280ae2cfc032ba19e: Deleting consensus metadata
I20251026 21:06:24.512370 6182 raft_consensus.cc:2272] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:24.512995 5516 catalog_manager.cc:4985] TS 8a4161c48ab347f280ae2cfc032ba19e (127.3.175.2:41147): tablet cfaf7af09d474c8c836cf0dbcda876eb (table test-table [id=d60fe7b19d62492783042299e20118dd]) successfully deleted
I20251026 21:06:24.513055 6182 ts_tablet_manager.cc:1916] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:24.514976 6182 ts_tablet_manager.cc:1929] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 21:06:24.515053 6182 log.cc:1199] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/cfaf7af09d474c8c836cf0dbcda876eb
I20251026 21:06:24.515435 6182 ts_tablet_manager.cc:1950] T cfaf7af09d474c8c836cf0dbcda876eb P ea99d00a02e14e5f88b87dd52416bdcc: Deleting consensus metadata
W20251026 21:06:24.515478 5519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.175.1:41291 (error 108)
I20251026 21:06:24.515648 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5748
I20251026 21:06:24.516803 5517 catalog_manager.cc:4985] TS ea99d00a02e14e5f88b87dd52416bdcc (127.3.175.3:41427): tablet cfaf7af09d474c8c836cf0dbcda876eb (table test-table [id=d60fe7b19d62492783042299e20118dd]) successfully deleted
I20251026 21:06:24.517252 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:24.516773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:41291 (local address 127.0.0.1:46858)
1026 21:06:24.517114 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:24.517169 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":95,"threads_started":1}
W20251026 21:06:24.517375 5519 catalog_manager.cc:4712] TS 546e3341248e4386ade22da1745d87f9 (127.3.175.1:41291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfaf7af09d474c8c836cf0dbcda876eb: Network error: Client connection negotiation failed: client connection to 127.3.175.1:41291: connect: Connection refused (error 111)
I20251026 21:06:24.522625 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5890
I20251026 21:06:24.529557 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 5500
2025-10-26T21:06:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7615 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:24 dist-test-slave-th6x krb5kdc[6187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:24 dist-test-slave-th6x krb5kdc[6187](info): set up 2 sockets
Oct 26 21:06:24 dist-test-slave-th6x krb5kdc[6187](info): commencing operation
krb5kdc: starting...
W20251026 21:06:26.511905 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:26 dist-test-slave-th6x krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:26.654420 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:36205
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:26.766458 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:26.766754 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:26.766846 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:26.770368 6203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:26.770459 6203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:26.770496 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:26.770530 6203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:26.770574 6203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:26.775161 6203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:26.776408 6203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:26.777333 6203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:26.786127 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:26.786127 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:26.786285 6203 server_base.cc:1047] running on GCE node
W20251026 21:06:26.786127 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:26.787110 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:26.788195 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:26.789439 6203 hybrid_clock.cc:648] HybridClock initialized: now 1761512786789429 us; error 30 us; skew 500 ppm
Oct 26 21:06:26 dist-test-slave-th6x krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:26.792449 6203 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:26.793619 6203 webserver.cc:492] Webserver started at http://127.3.175.62:45249/ using document root <none> and password file <none>
I20251026 21:06:26.794217 6203 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:26.794276 6203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:26.794471 6203 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:26.796368 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "2abca997ce5349b1ba9fe2169bfe9ef8"
format_stamp: "Formatted at 2025-10-26 21:06:26 on dist-test-slave-th6x"
server_key: "77df665b78105646f9e7296211c7a4ce"
server_key_iv: "33937d9b2d4d3446e8055018c5c42b97"
server_key_version: "encryptionkey@0"
I20251026 21:06:26.796882 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "2abca997ce5349b1ba9fe2169bfe9ef8"
format_stamp: "Formatted at 2025-10-26 21:06:26 on dist-test-slave-th6x"
server_key: "77df665b78105646f9e7296211c7a4ce"
server_key_iv: "33937d9b2d4d3446e8055018c5c42b97"
server_key_version: "encryptionkey@0"
I20251026 21:06:26.800493 6203 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:26.803005 6218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:26.804163 6203 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:26.804304 6203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "2abca997ce5349b1ba9fe2169bfe9ef8"
format_stamp: "Formatted at 2025-10-26 21:06:26 on dist-test-slave-th6x"
server_key: "77df665b78105646f9e7296211c7a4ce"
server_key_iv: "33937d9b2d4d3446e8055018c5c42b97"
server_key_version: "encryptionkey@0"
I20251026 21:06:26.804409 6203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:26.825464 6203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:26.828892 6203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:26.829088 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:26.836282 6203 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:36205
I20251026 21:06:26.836288 6270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:36205 every 8 connection(s)
I20251026 21:06:26.837409 6203 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:26.840461 6271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:26.841084 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6203
I20251026 21:06:26.841214 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:26.841450 3772 external_mini_cluster.cc:1468] Setting key 5df54c71523a7c6cd3cd03483bed8ee4
I20251026 21:06:26.847730 6271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8: Bootstrap starting.
Oct 26 21:06:26 dist-test-slave-th6x krb5kdc[6187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:26.850464 6271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:26.851382 6271 log.cc:826] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:26.853631 6271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8: No bootstrap required, opened a new log
I20251026 21:06:26.855221 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:26.842806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.3.175.62:36205)
1026 21:06:26.843269 (+ 463us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:26.843303 (+ 34us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:26.843347 (+ 44us) server_negotiation.cc:408] Connection header received
1026 21:06:26.844044 (+ 697us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:26.844090 (+ 46us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:26.844512 (+ 422us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:26.844897 (+ 385us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:26.845765 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:26.846930 (+ 1165us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:26.847687 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:26.847964 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:26.850275 (+ 2311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:26.850297 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:26.850313 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:26.850364 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:26.852406 (+ 2042us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:26.852886 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:26.852891 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:26.852897 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:26.853000 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:26.853234 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:26.853237 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:26.853239 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:26.853707 (+ 468us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:26.853898 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:26.854181 (+ 283us) server_negotiation.cc:300] Negotiation successful
1026 21:06:26.854446 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20251026 21:06:26.856961 6271 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } }
I20251026 21:06:26.857163 6271 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:26.857208 6271 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abca997ce5349b1ba9fe2169bfe9ef8, State: Initialized, Role: FOLLOWER
I20251026 21:06:26.857654 6271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } }
I20251026 21:06:26.857785 6271 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:26.857849 6271 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:26.857939 6271 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:26.858810 6271 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } }
I20251026 21:06:26.859154 6271 leader_election.cc:304] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abca997ce5349b1ba9fe2169bfe9ef8; no voters:
I20251026 21:06:26.859462 6271 leader_election.cc:290] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:26.859601 6276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:26.860159 6276 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [term 1 LEADER]: Becoming Leader. State: Replica: 2abca997ce5349b1ba9fe2169bfe9ef8, State: Running, Role: LEADER
I20251026 21:06:26.860616 6271 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:26.860749 6276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } }
I20251026 21:06:26.862329 6277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } } }
I20251026 21:06:26.862358 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abca997ce5349b1ba9fe2169bfe9ef8. Latest consensus state: current_term: 1 leader_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abca997ce5349b1ba9fe2169bfe9ef8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 36205 } } }
I20251026 21:06:26.862447 6277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:26.862466 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:26.862798 6286 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:26.865924 6286 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:26.872056 6286 catalog_manager.cc:1357] Generated new cluster ID: fc2317e19d264abbad2dcce3edf0316f
I20251026 21:06:26.872150 6286 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:26.887190 6286 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:26.888309 6286 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:26.899780 6286 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2abca997ce5349b1ba9fe2169bfe9ef8: Generated new TSK 0
I20251026 21:06:26.900642 6286 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:26.963907 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36205
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:27.075685 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:27.075944 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:27.076030 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:27.079416 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:27.079497 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:27.079610 6299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:27.084031 6299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:27.085193 6299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:27.086086 6299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:27.092895 6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:27.092902 6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:27.092890 6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:27.093042 6299 server_base.cc:1047] running on GCE node
I20251026 21:06:27.093546 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:27.094075 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:27.095245 6299 hybrid_clock.cc:648] HybridClock initialized: now 1761512787095229 us; error 32 us; skew 500 ppm
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:27.098026 6299 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:27.099114 6299 webserver.cc:492] Webserver started at http://127.3.175.1:35997/ using document root <none> and password file <none>
I20251026 21:06:27.099725 6299 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:27.099781 6299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:27.099989 6299 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:27.101701 6299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "0d4c97bd3f1c47fcb0458ef7ee667654"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "536e019b64f56706c35b6161fc171c94"
server_key_iv: "cb13e71c8195f7978da32624fa4f3466"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.102200 6299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "0d4c97bd3f1c47fcb0458ef7ee667654"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "536e019b64f56706c35b6161fc171c94"
server_key_iv: "cb13e71c8195f7978da32624fa4f3466"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.105712 6299 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:27.108098 6314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.109207 6299 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:27.109328 6299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "0d4c97bd3f1c47fcb0458ef7ee667654"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "536e019b64f56706c35b6161fc171c94"
server_key_iv: "cb13e71c8195f7978da32624fa4f3466"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.109426 6299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:27.133455 6299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:27.136796 6299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:27.137003 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:27.137578 6299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:27.138491 6299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:27.138545 6299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.138609 6299 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:27.138643 6299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.148079 6299 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:38473
I20251026 21:06:27.148118 6427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:38473 every 8 connection(s)
I20251026 21:06:27.149070 6299 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:27.150333 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6299
I20251026 21:06:27.150460 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:27.150705 3772 external_mini_cluster.cc:1468] Setting key 79442bb14edf4d2ce9714b4bd63d36be
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:27.163412 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.151010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:60553 (local address 127.3.175.62:36205)
1026 21:06:27.151151 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:27.151155 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:27.151967 (+ 812us) server_negotiation.cc:408] Connection header received
1026 21:06:27.152968 (+ 1001us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:27.152974 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:27.153041 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:27.153133 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:27.154971 (+ 1838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.155866 (+ 895us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.156542 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.156805 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.159446 (+ 2641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:27.159471 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:27.159474 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:27.159503 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:27.161138 (+ 1635us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.161682 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.161685 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.161686 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.161739 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.161977 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.161979 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.161980 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.162205 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:27.162309 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:27.162959 (+ 650us) server_negotiation.cc:300] Negotiation successful
1026 21:06:27.163119 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251026 21:06:27.163969 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.151353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36205 (local address 127.3.175.1:60553)
1026 21:06:27.151830 (+ 477us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:27.151875 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:27.152756 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:27.153325 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:27.153338 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:27.153811 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:27.154807 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.154824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.156002 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.156006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:27.156417 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.156424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.156613 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.157304 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:27.157332 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:27.159230 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:27.161261 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.161266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.161277 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.161578 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.161810 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.161812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.161814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.161902 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.162330 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:27.162340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:27.162673 (+ 333us) client_negotiation.cc:769] Sending connection context
1026 21:06:27.162887 (+ 214us) client_negotiation.cc:240] Negotiation successful
1026 21:06:27.163154 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20251026 21:06:27.165261 6428 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36205
I20251026 21:06:27.165544 6428 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:27.166270 6428 heartbeater.cc:507] Master 127.3.175.62:36205 requested a full tablet report, sending...
I20251026 21:06:27.168181 6235 ts_manager.cc:194] Registered new tserver with Master: 0d4c97bd3f1c47fcb0458ef7ee667654 (127.3.175.1:38473)
I20251026 21:06:27.169974 6235 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:60553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:27.202032 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36205
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:27.312834 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:27.313086 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:27.313144 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:27.316519 6435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:27.316594 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:27.316692 6435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:27.321033 6435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:27.322134 6435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:27.322963 6435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:27.329710 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:27.329792 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:27.329885 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:27.329792 6435 server_base.cc:1047] running on GCE node
I20251026 21:06:27.330348 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:27.330942 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:27.332118 6435 hybrid_clock.cc:648] HybridClock initialized: now 1761512787332103 us; error 31 us; skew 500 ppm
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:27.334944 6435 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:27.336177 6435 webserver.cc:492] Webserver started at http://127.3.175.2:35341/ using document root <none> and password file <none>
I20251026 21:06:27.336745 6435 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:27.336796 6435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:27.337004 6435 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:27.338778 6435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "caec0f7db79e4c9e8e892d03505a0cfd"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "e1182a8022a45d12524e7f4d987ae6dc"
server_key_iv: "611bd3254125350cc639589eeed095ca"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.339260 6435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "caec0f7db79e4c9e8e892d03505a0cfd"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "e1182a8022a45d12524e7f4d987ae6dc"
server_key_iv: "611bd3254125350cc639589eeed095ca"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.342873 6435 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:27.345291 6450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.346386 6435 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:27.346493 6435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "caec0f7db79e4c9e8e892d03505a0cfd"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "e1182a8022a45d12524e7f4d987ae6dc"
server_key_iv: "611bd3254125350cc639589eeed095ca"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.346606 6435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:27.386302 6435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:27.389583 6435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:27.389788 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:27.390401 6435 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:27.391315 6435 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:27.391363 6435 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.391412 6435 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:27.391443 6435 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.400707 6435 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:41843
I20251026 21:06:27.400730 6563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:41843 every 8 connection(s)
I20251026 21:06:27.401769 6435 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:27.408998 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6435
I20251026 21:06:27.409112 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:27.409343 3772 external_mini_cluster.cc:1468] Setting key cb3200aa088e773878645567b250ccf6
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:27.415162 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.403585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:34497 (local address 127.3.175.62:36205)
1026 21:06:27.403744 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:27.403749 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:27.404561 (+ 812us) server_negotiation.cc:408] Connection header received
1026 21:06:27.405382 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:27.405386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:27.405437 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:27.405513 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:27.407188 (+ 1675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.407800 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.408488 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.408716 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.411130 (+ 2414us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:27.411152 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:27.411158 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:27.411195 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:27.412850 (+ 1655us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.413482 (+ 632us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.413485 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.413487 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.413535 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.413942 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.413946 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.413948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.414128 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:27.414235 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:27.414921 (+ 686us) server_negotiation.cc:300] Negotiation successful
1026 21:06:27.415039 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:06:27.416285 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.403914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36205 (local address 127.3.175.2:34497)
1026 21:06:27.404406 (+ 492us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:27.404446 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:27.405200 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:27.405659 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:27.405667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:27.406084 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:27.407031 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.407047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.407929 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.407932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:27.408348 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.408356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.408541 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.409276 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:27.409303 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:27.410961 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:27.412973 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.412997 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.413014 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.413388 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.413742 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.413746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.413748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.413845 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.414266 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:27.414278 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:27.414619 (+ 341us) client_negotiation.cc:769] Sending connection context
1026 21:06:27.414876 (+ 257us) client_negotiation.cc:240] Negotiation successful
1026 21:06:27.415221 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":117,"threads_started":1}
I20251026 21:06:27.417582 6564 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36205
I20251026 21:06:27.417851 6564 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:27.418437 6564 heartbeater.cc:507] Master 127.3.175.62:36205 requested a full tablet report, sending...
I20251026 21:06:27.419530 6235 ts_manager.cc:194] Registered new tserver with Master: caec0f7db79e4c9e8e892d03505a0cfd (127.3.175.2:41843)
I20251026 21:06:27.420109 6235 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:34497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:27.460178 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:36205
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:27.571412 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:27.571731 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:27.571802 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:27.575047 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:27.575119 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:27.575201 6571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:27.579607 6571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:27.580806 6571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:27.581702 6571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:27.588953 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:27.589027 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:27.589073 6571 server_base.cc:1047] running on GCE node
W20251026 21:06:27.588953 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:27.589605 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:27.590191 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:27.591377 6571 hybrid_clock.cc:648] HybridClock initialized: now 1761512787591353 us; error 37 us; skew 500 ppm
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:27.594264 6571 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:27.595503 6571 webserver.cc:492] Webserver started at http://127.3.175.3:46517/ using document root <none> and password file <none>
I20251026 21:06:27.596107 6571 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:27.596163 6571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:27.596354 6571 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:27.598130 6571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "cd54cda6e4ac4b2d99486538cf9edfd0"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "3d1f67de0d52dbfb341ef6b435dd833d"
server_key_iv: "8ebac1655c7246b5026b5bbe1fcfe1de"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.598618 6571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "cd54cda6e4ac4b2d99486538cf9edfd0"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "3d1f67de0d52dbfb341ef6b435dd833d"
server_key_iv: "8ebac1655c7246b5026b5bbe1fcfe1de"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.602269 6571 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:27.604738 6586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.605823 6571 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:27.605942 6571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "cd54cda6e4ac4b2d99486538cf9edfd0"
format_stamp: "Formatted at 2025-10-26 21:06:27 on dist-test-slave-th6x"
server_key: "3d1f67de0d52dbfb341ef6b435dd833d"
server_key_iv: "8ebac1655c7246b5026b5bbe1fcfe1de"
server_key_version: "encryptionkey@0"
I20251026 21:06:27.606045 6571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:27.619980 6571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:27.627542 6571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:27.627761 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:27.628367 6571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:27.629333 6571 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:27.629386 6571 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.629452 6571 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:27.629482 6571 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:27.641135 6571 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:41749
I20251026 21:06:27.641148 6699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:41749 every 8 connection(s)
I20251026 21:06:27.642329 6571 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:27.646715 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6571
I20251026 21:06:27.646842 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:27.647084 3772 external_mini_cluster.cc:1468] Setting key 17354df42778f1d11e34dc9e1ff7a917
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:27.655882 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.644283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:49007 (local address 127.3.175.62:36205)
1026 21:06:27.644447 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:27.644452 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:27.645210 (+ 758us) server_negotiation.cc:408] Connection header received
1026 21:06:27.646072 (+ 862us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:27.646076 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:27.646132 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:27.646215 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:27.648197 (+ 1982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.648926 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.649749 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.649887 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.652347 (+ 2460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:27.652368 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:27.652371 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:27.652402 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:27.653880 (+ 1478us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.654393 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.654396 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.654397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.654447 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:27.654787 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:27.654790 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:27.654792 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:27.654964 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:27.655057 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:27.655610 (+ 553us) server_negotiation.cc:300] Negotiation successful
1026 21:06:27.655729 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:06:27.656553 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:27.644586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:36205 (local address 127.3.175.3:49007)
1026 21:06:27.645065 (+ 479us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:27.645103 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:27.645845 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:27.646371 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:27.646379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:27.646792 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:27.648010 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.648030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.649064 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:27.649068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:27.649575 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:27.649584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:27.649774 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:27.650405 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:27.650431 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:27.652170 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:27.654017 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.654023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.654034 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.654285 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.654528 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:27.654530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:27.654532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:27.654617 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:27.655064 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:27.655069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:27.655373 (+ 304us) client_negotiation.cc:769] Sending connection context
1026 21:06:27.655569 (+ 196us) client_negotiation.cc:240] Negotiation successful
1026 21:06:27.655806 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20251026 21:06:27.657763 6700 heartbeater.cc:344] Connected to a master server at 127.3.175.62:36205
I20251026 21:06:27.658034 6700 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:27.658579 6700 heartbeater.cc:507] Master 127.3.175.62:36205 requested a full tablet report, sending...
I20251026 21:06:27.659657 6235 ts_manager.cc:194] Registered new tserver with Master: cd54cda6e4ac4b2d99486538cf9edfd0 (127.3.175.3:41749)
I20251026 21:06:27.660223 6235 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:49007
I20251026 21:06:27.661211 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 21:06:27.673808 6274 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:27.672417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.3.175.62:36205)
1026 21:06:27.672583 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:27.672587 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:27.672644 (+ 57us) server_negotiation.cc:408] Connection header received
1026 21:06:27.672807 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:27.672810 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:27.672858 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:27.672932 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:27.673646 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38582: BlockingRecv error: recv got EOF from 127.0.0.1:38582 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20251026 21:06:27.675699 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6299
I20251026 21:06:27.681129 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6435
I20251026 21:06:27.687023 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6571
I20251026 21:06:27.692869 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6203
2025-10-26T21:06:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3161 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6712](info): set up 2 sockets
Oct 26 21:06:27 dist-test-slave-th6x krb5kdc[6712](info): commencing operation
krb5kdc: starting...
W20251026 21:06:29.704176 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:29 dist-test-slave-th6x krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:29.847640 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44061
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:29.956768 6728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:29.957064 6728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:29.957165 6728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:29.960601 6728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:29.960683 6728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:29.960723 6728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:29.960757 6728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:29.960798 6728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:29.965397 6728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:29.966609 6728 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:29.967557 6728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:29.973521 6733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:29.973717 6728 server_base.cc:1047] running on GCE node
W20251026 21:06:29.973506 6734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:29.973513 6736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:29.974391 6728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:29.975425 6728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:29.976584 6728 hybrid_clock.cc:648] HybridClock initialized: now 1761512789976571 us; error 33 us; skew 500 ppm
Oct 26 21:06:29 dist-test-slave-th6x krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:29.979655 6728 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:29.980928 6728 webserver.cc:492] Webserver started at http://127.3.175.62:45943/ using document root <none> and password file <none>
I20251026 21:06:29.981544 6728 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:29.981598 6728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:29.981812 6728 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:29.983667 6728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "3b22543f7b864fbf97364b0eecd0319a"
format_stamp: "Formatted at 2025-10-26 21:06:29 on dist-test-slave-th6x"
server_key: "b4ed169097bdf1c3d8a00e91ec185dfc"
server_key_iv: "ea3dac7400d85f5f9106a11f15a88cec"
server_key_version: "encryptionkey@0"
I20251026 21:06:29.984167 6728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "3b22543f7b864fbf97364b0eecd0319a"
format_stamp: "Formatted at 2025-10-26 21:06:29 on dist-test-slave-th6x"
server_key: "b4ed169097bdf1c3d8a00e91ec185dfc"
server_key_iv: "ea3dac7400d85f5f9106a11f15a88cec"
server_key_version: "encryptionkey@0"
I20251026 21:06:29.987697 6728 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:29.990103 6743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:29.991148 6728 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:29.991266 6728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "3b22543f7b864fbf97364b0eecd0319a"
format_stamp: "Formatted at 2025-10-26 21:06:29 on dist-test-slave-th6x"
server_key: "b4ed169097bdf1c3d8a00e91ec185dfc"
server_key_iv: "ea3dac7400d85f5f9106a11f15a88cec"
server_key_version: "encryptionkey@0"
I20251026 21:06:29.991415 6728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:30.001729 6728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:30.004730 6728 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:30.004947 6728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:30.012552 6795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:44061 every 8 connection(s)
I20251026 21:06:30.012558 6728 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:44061
I20251026 21:06:30.013686 6728 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:30.016669 6796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:30.022464 6796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a: Bootstrap starting.
I20251026 21:06:30.023624 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6728
I20251026 21:06:30.023763 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:30.024006 3772 external_mini_cluster.cc:1468] Setting key 9ec73cbabd97dbe9f28a24bbc63277d6
I20251026 21:06:30.025360 6796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:30.026150 6796 log.cc:826] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:30.028414 6796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a: No bootstrap required, opened a new log
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:30.031905 6796 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } }
I20251026 21:06:30.032176 6796 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:30.032244 6796 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b22543f7b864fbf97364b0eecd0319a, State: Initialized, Role: FOLLOWER
I20251026 21:06:30.032727 6796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } }
I20251026 21:06:30.032864 6796 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:30.032940 6796 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:30.033037 6796 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:30.033864 6796 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } }
I20251026 21:06:30.034231 6796 leader_election.cc:304] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b22543f7b864fbf97364b0eecd0319a; no voters:
I20251026 21:06:30.034536 6796 leader_election.cc:290] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:30.034659 6801 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:30.034896 6801 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [term 1 LEADER]: Becoming Leader. State: Replica: 3b22543f7b864fbf97364b0eecd0319a, State: Running, Role: LEADER
I20251026 21:06:30.035255 6801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } }
I20251026 21:06:30.036096 6796 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:30.036955 6802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b22543f7b864fbf97364b0eecd0319a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } } }
I20251026 21:06:30.037093 6802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:30.037392 6803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b22543f7b864fbf97364b0eecd0319a. Latest consensus state: current_term: 1 leader_uuid: "3b22543f7b864fbf97364b0eecd0319a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b22543f7b864fbf97364b0eecd0319a" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44061 } } }
I20251026 21:06:30.037480 6803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:30.037755 6806 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:30.038784 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.025089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.3.175.62:44061)
1026 21:06:30.025508 (+ 419us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:30.025521 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:30.025560 (+ 39us) server_negotiation.cc:408] Connection header received
1026 21:06:30.026176 (+ 616us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:30.026194 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:30.026548 (+ 354us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:30.026947 (+ 399us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:30.028064 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.028872 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.029734 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.029941 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.032550 (+ 2609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:30.032576 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:30.032588 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:30.032632 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:30.034682 (+ 2050us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.035306 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.035314 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.035320 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.035438 (+ 118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.035715 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.035718 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.035720 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.036157 (+ 437us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:30.036439 (+ 282us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:30.037392 (+ 953us) server_negotiation.cc:300] Negotiation successful
1026 21:06:30.037856 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20251026 21:06:30.041476 6806 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:30.047411 6806 catalog_manager.cc:1357] Generated new cluster ID: cdcd4e0a44564ff09858b1a3a160c1b1
I20251026 21:06:30.047506 6806 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:30.057633 6806 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:30.058950 6806 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:30.068419 6806 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3b22543f7b864fbf97364b0eecd0319a: Generated new TSK 0
I20251026 21:06:30.069495 6806 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:30.134258 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44061
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:30.250555 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:30.250834 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:30.250898 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:30.254436 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:30.254539 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:30.254634 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:30.259480 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:30.260705 6824 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:30.261644 6824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:30.268729 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:30.268678 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:30.268678 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:30.269097 6824 server_base.cc:1047] running on GCE node
I20251026 21:06:30.269465 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:30.270092 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:30.271250 6824 hybrid_clock.cc:648] HybridClock initialized: now 1761512790271227 us; error 37 us; skew 500 ppm
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:30.274104 6824 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:30.275246 6824 webserver.cc:492] Webserver started at http://127.3.175.1:40973/ using document root <none> and password file <none>
I20251026 21:06:30.275856 6824 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:30.275913 6824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:30.276093 6824 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:30.277866 6824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "5a06dc050ed9461ba8e23cc5cd90ce46"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "ab4ca55fb57c56c0f16461d3987a5bc3"
server_key_iv: "cdd77538b8e1f1c8122169d39c9401db"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.278347 6824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "5a06dc050ed9461ba8e23cc5cd90ce46"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "ab4ca55fb57c56c0f16461d3987a5bc3"
server_key_iv: "cdd77538b8e1f1c8122169d39c9401db"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.282150 6824 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20251026 21:06:30.284682 6839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.286010 6824 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:30.286171 6824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "5a06dc050ed9461ba8e23cc5cd90ce46"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "ab4ca55fb57c56c0f16461d3987a5bc3"
server_key_iv: "cdd77538b8e1f1c8122169d39c9401db"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.286300 6824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:30.298987 6824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:30.302201 6824 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:30.302371 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:30.302982 6824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:30.303972 6824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:30.304021 6824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.304071 6824 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:30.304104 6824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.313055 6824 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:44513
I20251026 21:06:30.313081 6952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:44513 every 8 connection(s)
I20251026 21:06:30.314066 6824 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:30.320958 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6824
I20251026 21:06:30.321079 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:30.321329 3772 external_mini_cluster.cc:1468] Setting key 81668f759f567ceadb4e4bf9b25071e9
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:30.328059 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.315931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:56481 (local address 127.3.175.62:44061)
1026 21:06:30.316113 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:30.316117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:30.316863 (+ 746us) server_negotiation.cc:408] Connection header received
1026 21:06:30.317788 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:30.317793 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:30.317861 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:30.317959 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:30.319862 (+ 1903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.320392 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.321146 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.321351 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.323755 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:30.323775 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:30.323777 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:30.323805 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:30.325715 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.326311 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.326318 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.326323 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.326391 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.326677 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.326683 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.326687 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.326931 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:30.327069 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:30.327613 (+ 544us) server_negotiation.cc:300] Negotiation successful
1026 21:06:30.327799 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251026 21:06:30.328521 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.316174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44061 (local address 127.3.175.1:56481)
1026 21:06:30.316687 (+ 513us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:30.316729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:30.317567 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:30.318143 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:30.318153 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:30.318600 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:30.319669 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.319687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.320547 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.320550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:30.321004 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.321013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.321260 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.321984 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:30.322012 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:30.323591 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:30.325859 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.325864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.325874 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.326173 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.326491 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.326494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.326495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.326587 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.327046 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:30.327052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:30.327339 (+ 287us) client_negotiation.cc:769] Sending connection context
1026 21:06:30.327534 (+ 195us) client_negotiation.cc:240] Negotiation successful
1026 21:06:30.327780 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
I20251026 21:06:30.329845 6953 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44061
I20251026 21:06:30.330108 6953 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:30.330668 6953 heartbeater.cc:507] Master 127.3.175.62:44061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:06:30.332372 6760 ts_manager.cc:194] Registered new tserver with Master: 5a06dc050ed9461ba8e23cc5cd90ce46 (127.3.175.1:44513)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:30.333747 6760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:56481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:30.370277 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44061
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:30.484077 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:30.484357 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:30.484453 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:30.487905 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:30.487994 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:30.488127 6960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:30.492666 6960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:30.493830 6960 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:30.494706 6960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:30.501453 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:30.501453 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:30.501473 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:30.501724 6960 server_base.cc:1047] running on GCE node
I20251026 21:06:30.502151 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:30.502746 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:30.503909 6960 hybrid_clock.cc:648] HybridClock initialized: now 1761512790503882 us; error 40 us; skew 500 ppm
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:30.506647 6960 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:30.507745 6960 webserver.cc:492] Webserver started at http://127.3.175.2:40597/ using document root <none> and password file <none>
I20251026 21:06:30.508282 6960 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:30.508330 6960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:30.508522 6960 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:30.510277 6960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "490f176ee02b46cf8adefe14132c6ad8"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "327f8df8d7f9dcee473859a083068b69"
server_key_iv: "8971b111263b7d083dc5550e5c9e1609"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.510716 6960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "490f176ee02b46cf8adefe14132c6ad8"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "327f8df8d7f9dcee473859a083068b69"
server_key_iv: "8971b111263b7d083dc5550e5c9e1609"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.514235 6960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:30.516552 6975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.517622 6960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:30.517792 6960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "490f176ee02b46cf8adefe14132c6ad8"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "327f8df8d7f9dcee473859a083068b69"
server_key_iv: "8971b111263b7d083dc5550e5c9e1609"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.517884 6960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:30.537248 6960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:30.540390 6960 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:30.540597 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:30.541172 6960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:30.542107 6960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:30.542160 6960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.542222 6960 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:30.542249 6960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.551789 6960 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:39147
I20251026 21:06:30.551841 7088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:39147 every 8 connection(s)
I20251026 21:06:30.552796 6960 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:30.556540 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 6960
I20251026 21:06:30.556682 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:30.556931 3772 external_mini_cluster.cc:1468] Setting key 1855a7d2fdd3f6c46d12738aa92ca143
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:30.567098 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.554638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:50321 (local address 127.3.175.62:44061)
1026 21:06:30.554799 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:30.554802 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:30.555679 (+ 877us) server_negotiation.cc:408] Connection header received
1026 21:06:30.556714 (+ 1035us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:30.556718 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:30.556787 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:30.556873 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:30.558938 (+ 2065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.559489 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.560423 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.560650 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.562994 (+ 2344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:30.563024 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:30.563031 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:30.563072 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:30.564968 (+ 1896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.565477 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.565480 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.565482 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.565537 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.565893 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.565898 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.565901 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.566121 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:30.566238 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:30.566861 (+ 623us) server_negotiation.cc:300] Negotiation successful
1026 21:06:30.566970 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251026 21:06:30.567939 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.554996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44061 (local address 127.3.175.2:50321)
1026 21:06:30.555525 (+ 529us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:30.555564 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:30.556435 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:30.557084 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:30.557096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:30.557595 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:30.558782 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.558801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.559611 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.559618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:30.560289 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.560302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.560504 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.561164 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:30.561196 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:30.562819 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:30.565095 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.565100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.565111 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.565372 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.565673 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.565676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.565678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.565786 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.566247 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:30.566254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:30.566500 (+ 246us) client_negotiation.cc:769] Sending connection context
1026 21:06:30.566801 (+ 301us) client_negotiation.cc:240] Negotiation successful
1026 21:06:30.567075 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:30.569267 7089 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44061
I20251026 21:06:30.569517 7089 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:30.570087 7089 heartbeater.cc:507] Master 127.3.175.62:44061 requested a full tablet report, sending...
I20251026 21:06:30.571245 6760 ts_manager.cc:194] Registered new tserver with Master: 490f176ee02b46cf8adefe14132c6ad8 (127.3.175.2:39147)
I20251026 21:06:30.571884 6760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:50321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:30.610014 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:44061
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:30.723250 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:30.723554 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:30.723644 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:30.727048 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:30.727115 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:30.727213 7096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:30.731379 7096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:30.732447 7096 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:30.733294 7096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:30.740006 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:30.739984 7102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:30.740090 7096 server_base.cc:1047] running on GCE node
W20251026 21:06:30.739986 7101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:30.740602 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:30.741174 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:30.742332 7096 hybrid_clock.cc:648] HybridClock initialized: now 1761512790742314 us; error 32 us; skew 500 ppm
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:30.745182 7096 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:30.746410 7096 webserver.cc:492] Webserver started at http://127.3.175.3:41977/ using document root <none> and password file <none>
I20251026 21:06:30.746999 7096 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:30.747057 7096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:30.747246 7096 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:30.749027 7096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "67b87382d949482c92681c7382003ff6"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "a5e46183555d9b4b05dc8e4623354947"
server_key_iv: "247905827a15268165b5be62898bb5c7"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.749532 7096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "67b87382d949482c92681c7382003ff6"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "a5e46183555d9b4b05dc8e4623354947"
server_key_iv: "247905827a15268165b5be62898bb5c7"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.753235 7096 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:30.755615 7111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.756852 7096 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:06:30.757109 7096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "67b87382d949482c92681c7382003ff6"
format_stamp: "Formatted at 2025-10-26 21:06:30 on dist-test-slave-th6x"
server_key: "a5e46183555d9b4b05dc8e4623354947"
server_key_iv: "247905827a15268165b5be62898bb5c7"
server_key_version: "encryptionkey@0"
I20251026 21:06:30.757331 7096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:30.777470 7096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:30.780872 7096 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:30.781064 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:30.781666 7096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:30.782598 7096 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:30.782658 7096 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.782723 7096 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:30.782753 7096 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:30.793766 7096 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:36975
I20251026 21:06:30.793833 7224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:36975 every 8 connection(s)
I20251026 21:06:30.794811 7096 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:30.796427 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7096
I20251026 21:06:30.796547 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:30.796806 3772 external_mini_cluster.cc:1468] Setting key 8fce4ba97f77b1612ff6a46c091f636d
Oct 26 21:06:30 dist-test-slave-th6x krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:30.810276 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.798276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:38453 (local address 127.3.175.62:44061)
1026 21:06:30.798403 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:30.798406 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:30.798419 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:06:30.799246 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:30.799251 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:30.799348 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:30.799453 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:30.801232 (+ 1779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.802212 (+ 980us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.803000 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.803241 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.806018 (+ 2777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:30.806061 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:30.806066 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:30.806127 (+ 61us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:30.808017 (+ 1890us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.808549 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.808554 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.808558 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.808622 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:30.808932 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:30.808937 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:30.808942 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:30.809177 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:30.809285 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:30.809958 (+ 673us) server_negotiation.cc:300] Negotiation successful
1026 21:06:30.810099 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 21:06:30.811239 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:30.797175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44061 (local address 127.3.175.3:38453)
1026 21:06:30.798052 (+ 877us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:30.798101 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:30.799042 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:30.799587 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:30.799595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:30.800061 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:30.801077 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.801093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.802383 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:30.802387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:30.802877 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:30.802886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:30.803099 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:30.803990 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:30.804018 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:30.805831 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:30.808165 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.808171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.808181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.808437 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.808724 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:30.808745 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:30.808747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:30.808842 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:30.809311 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:30.809319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:30.809627 (+ 308us) client_negotiation.cc:769] Sending connection context
1026 21:06:30.809913 (+ 286us) client_negotiation.cc:240] Negotiation successful
1026 21:06:30.810216 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20251026 21:06:30.812398 7225 heartbeater.cc:344] Connected to a master server at 127.3.175.62:44061
I20251026 21:06:30.812704 7225 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:30.813475 7225 heartbeater.cc:507] Master 127.3.175.62:44061 requested a full tablet report, sending...
I20251026 21:06:30.814780 6760 ts_manager.cc:194] Registered new tserver with Master: 67b87382d949482c92681c7382003ff6 (127.3.175.3:36975)
I20251026 21:06:30.815759 6760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:38453
I20251026 21:06:30.823565 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 21:06:31.033862 6799 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:31.030041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.3.175.62:44061)
1026 21:06:31.030207 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:31.030212 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:31.031201 (+ 989us) server_negotiation.cc:408] Connection header received
1026 21:06:31.032166 (+ 965us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:31.032170 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:31.032226 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:31.032317 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:31.033711 (+ 1394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40198: BlockingRecv error: recv got EOF from 127.0.0.1:40198 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20251026 21:06:31.035178 6799 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:31.034391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.3.175.62:44061)
1026 21:06:31.034504 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:31.034507 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:31.034519 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:06:31.034649 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:31.034651 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:31.034688 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:31.034758 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:31.035092 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20251026 21:06:31.036346 6799 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:31.035595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.3.175.62:44061)
1026 21:06:31.035708 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:31.035711 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:31.035724 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:06:31.035831 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:31.035834 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:31.035872 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:31.035934 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:31.036256 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40212: BlockingRecv error: recv got EOF from 127.0.0.1:40212 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20251026 21:06:31.040877 6799 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:31.039813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.3.175.62:44061)
1026 21:06:31.039966 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:31.039969 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:31.040031 (+ 62us) server_negotiation.cc:408] Connection header received
1026 21:06:31.040140 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:31.040142 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:31.040178 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:31.040240 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:31.040765 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40228: BlockingRecv error: recv got EOF from 127.0.0.1:40228 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20251026 21:06:31.049346 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6824
I20251026 21:06:31.055124 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6960
I20251026 21:06:31.061100 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7096
I20251026 21:06:31.066552 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 6728
2025-10-26T21:06:31Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3374 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:31 dist-test-slave-th6x krb5kdc[7248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:31 dist-test-slave-th6x krb5kdc[7248](info): set up 2 sockets
Oct 26 21:06:31 dist-test-slave-th6x krb5kdc[7248](info): commencing operation
krb5kdc: starting...
W20251026 21:06:33.051699 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.962s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:33.192340 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44497
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:33.303819 7264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:33.304073 7264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:33.304134 7264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:33.307507 7264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:33.307578 7264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:33.307601 7264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:33.307621 7264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:33.307638 7264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:33.312196 7264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:33.313417 7264 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:33.314316 7264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:33.320170 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:33.320165 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:33.320165 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:33.320374 7264 server_base.cc:1047] running on GCE node
I20251026 21:06:33.320932 7264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:33.321863 7264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:33.323024 7264 hybrid_clock.cc:648] HybridClock initialized: now 1761512793323012 us; error 29 us; skew 500 ppm
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512793, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:33.325984 7264 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:33.327490 7264 webserver.cc:492] Webserver started at http://127.3.175.62:44073/ using document root <none> and password file <none>
I20251026 21:06:33.328080 7264 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:33.328132 7264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:33.328305 7264 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:33.330065 7264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "b038e6028a94460a95e32c9752f6446b"
format_stamp: "Formatted at 2025-10-26 21:06:33 on dist-test-slave-th6x"
server_key: "3280f2f47eff0550b92e9e7804164bf5"
server_key_iv: "7a785df213e9fa6e35addf2e2683f7a2"
server_key_version: "encryptionkey@0"
I20251026 21:06:33.330536 7264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "b038e6028a94460a95e32c9752f6446b"
format_stamp: "Formatted at 2025-10-26 21:06:33 on dist-test-slave-th6x"
server_key: "3280f2f47eff0550b92e9e7804164bf5"
server_key_iv: "7a785df213e9fa6e35addf2e2683f7a2"
server_key_version: "encryptionkey@0"
I20251026 21:06:33.334354 7264 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251026 21:06:33.336781 7279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:33.337958 7264 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:33.338085 7264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "b038e6028a94460a95e32c9752f6446b"
format_stamp: "Formatted at 2025-10-26 21:06:33 on dist-test-slave-th6x"
server_key: "3280f2f47eff0550b92e9e7804164bf5"
server_key_iv: "7a785df213e9fa6e35addf2e2683f7a2"
server_key_version: "encryptionkey@0"
I20251026 21:06:33.338179 7264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:33.365713 7264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:33.373078 7264 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:33.373252 7264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:33.380862 7264 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:44497
I20251026 21:06:33.380967 7331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:44497 every 8 connection(s)
I20251026 21:06:33.381913 7264 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:33.384878 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:33.388796 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7264
I20251026 21:06:33.388932 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:33.389180 3772 external_mini_cluster.cc:1468] Setting key 18aad8de54d52f7a9304b4522e3c61df
I20251026 21:06:33.391213 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: Bootstrap starting.
I20251026 21:06:33.393464 7332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:33.394204 7332 log.cc:826] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:33.396160 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: No bootstrap required, opened a new log
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:33.399238 7332 raft_consensus.cc:359] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } }
I20251026 21:06:33.399489 7332 raft_consensus.cc:385] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:33.399567 7332 raft_consensus.cc:740] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b038e6028a94460a95e32c9752f6446b, State: Initialized, Role: FOLLOWER
I20251026 21:06:33.400024 7332 consensus_queue.cc:260] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } }
I20251026 21:06:33.400161 7332 raft_consensus.cc:399] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:33.400228 7332 raft_consensus.cc:493] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:33.400318 7332 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:33.401397 7332 raft_consensus.cc:515] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } }
I20251026 21:06:33.401823 7332 leader_election.cc:304] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b038e6028a94460a95e32c9752f6446b; no voters:
I20251026 21:06:33.402189 7332 leader_election.cc:290] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:33.402585 7337 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:33.402803 7337 raft_consensus.cc:697] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [term 1 LEADER]: Becoming Leader. State: Replica: b038e6028a94460a95e32c9752f6446b, State: Running, Role: LEADER
I20251026 21:06:33.402961 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:33.390547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.3.175.62:44497)
1026 21:06:33.391028 (+ 481us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:33.391038 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:33.391074 (+ 36us) server_negotiation.cc:408] Connection header received
1026 21:06:33.391678 (+ 604us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:33.391701 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:33.392027 (+ 326us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:33.392351 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:33.393344 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:33.394133 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:33.394856 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:33.395155 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:33.397789 (+ 2634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:33.397810 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:33.397822 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:33.397857 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:33.399949 (+ 2092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:33.400537 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:33.400543 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:33.400549 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:33.400649 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:33.400909 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:33.400913 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:33.400915 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:33.401399 (+ 484us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:33.401624 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:33.401885 (+ 261us) server_negotiation.cc:300] Negotiation successful
1026 21:06:33.402179 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
I20251026 21:06:33.403137 7337 consensus_queue.cc:237] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } }
I20251026 21:06:33.403429 7332 sys_catalog.cc:565] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:33.404690 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b038e6028a94460a95e32c9752f6446b. Latest consensus state: current_term: 1 leader_uuid: "b038e6028a94460a95e32c9752f6446b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } } }
I20251026 21:06:33.404826 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:33.404932 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b038e6028a94460a95e32c9752f6446b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b038e6028a94460a95e32c9752f6446b" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44497 } } }
I20251026 21:06:33.405014 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:33.406653 3772 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20251026 21:06:33.408712 7352 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 21:06:33.408802 7352 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 21:06:33.408864 7346 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:33.409888 7346 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:33.416630 7346 catalog_manager.cc:1357] Generated new cluster ID: 08789302f7544e1bbe7ebce5f4d4b1f1
I20251026 21:06:33.416713 7346 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:33.425005 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:33.420110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.3.175.62:44497)
1026 21:06:33.420270 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:33.420276 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:33.420381 (+ 105us) server_negotiation.cc:408] Connection header received
1026 21:06:33.420503 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:33.420506 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:33.420565 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:33.420662 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:33.421402 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:33.422163 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:33.422994 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:33.423203 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:33.423837 (+ 634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:33.423861 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:33.423865 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:33.424452 (+ 587us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:33.424488 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:33.424507 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:33.424600 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:33.424774 (+ 174us) server_negotiation.cc:300] Negotiation successful
1026 21:06:33.424836 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251026 21:06:33.427062 7346 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:33.428006 7346 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:33.437673 7346 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b038e6028a94460a95e32c9752f6446b: Generated new TSK 0
I20251026 21:06:33.438455 7346 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:06:33.446791 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7264
2025-10-26T21:06:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2377 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7365](info): set up 2 sockets
Oct 26 21:06:33 dist-test-slave-th6x krb5kdc[7365](info): commencing operation
krb5kdc: starting...
W20251026 21:06:35.447913 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:35 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:35.586382 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:42097
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:35.703666 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:35.703979 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:35.704074 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:35.707630 7381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:35.707746 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:35.707790 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:35.707849 7381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:35.707893 7381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:35.712888 7381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:35.714104 7381 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:35.715000 7381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:35.721276 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:35.721277 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:35.721303 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:35.721566 7381 server_base.cc:1047] running on GCE node
I20251026 21:06:35.722179 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:35.723201 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:35.724390 7381 hybrid_clock.cc:648] HybridClock initialized: now 1761512795724371 us; error 33 us; skew 500 ppm
Oct 26 21:06:35 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:35.727584 7381 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:35.728757 7381 webserver.cc:492] Webserver started at http://127.3.175.62:34721/ using document root <none> and password file <none>
I20251026 21:06:35.729341 7381 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:35.729394 7381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:35.729581 7381 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:35.731559 7381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2"
format_stamp: "Formatted at 2025-10-26 21:06:35 on dist-test-slave-th6x"
server_key: "c760d2f3aeb7c4329665202082d64ee7"
server_key_iv: "23493ccfdaf5ad4a7b41f74a91af9468"
server_key_version: "encryptionkey@0"
I20251026 21:06:35.732105 7381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2"
format_stamp: "Formatted at 2025-10-26 21:06:35 on dist-test-slave-th6x"
server_key: "c760d2f3aeb7c4329665202082d64ee7"
server_key_iv: "23493ccfdaf5ad4a7b41f74a91af9468"
server_key_version: "encryptionkey@0"
I20251026 21:06:35.735648 7381 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251026 21:06:35.738104 7396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:35.739180 7381 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:35.739326 7381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2"
format_stamp: "Formatted at 2025-10-26 21:06:35 on dist-test-slave-th6x"
server_key: "c760d2f3aeb7c4329665202082d64ee7"
server_key_iv: "23493ccfdaf5ad4a7b41f74a91af9468"
server_key_version: "encryptionkey@0"
I20251026 21:06:35.739428 7381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:35.761166 7381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:35.764721 7381 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:35.764948 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:35.772576 7381 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:42097
I20251026 21:06:35.772645 7448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:42097 every 8 connection(s)
I20251026 21:06:35.773691 7381 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:35.776640 7449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:35.782326 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: Bootstrap starting.
I20251026 21:06:35.782922 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7381
I20251026 21:06:35.783066 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:35.783350 3772 external_mini_cluster.cc:1468] Setting key ed4af8d9849dee18bc4f0a0aa8fc64cd
I20251026 21:06:35.785194 7449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:35.785912 7449 log.cc:826] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:35.788143 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: No bootstrap required, opened a new log
Oct 26 21:06:35 dist-test-slave-th6x krb5kdc[7365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:35.791646 7449 raft_consensus.cc:359] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } }
I20251026 21:06:35.791883 7449 raft_consensus.cc:385] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:35.791953 7449 raft_consensus.cc:740] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ef05fecf19496cb1e4fc6ccb7fe6c2, State: Initialized, Role: FOLLOWER
I20251026 21:06:35.792459 7449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } }
I20251026 21:06:35.792668 7449 raft_consensus.cc:399] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:35.792748 7449 raft_consensus.cc:493] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:35.792856 7449 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:35.793738 7449 raft_consensus.cc:515] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } }
I20251026 21:06:35.794104 7449 leader_election.cc:304] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ef05fecf19496cb1e4fc6ccb7fe6c2; no voters:
I20251026 21:06:35.794525 7449 leader_election.cc:290] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:35.794651 7454 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:35.794867 7454 raft_consensus.cc:697] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [term 1 LEADER]: Becoming Leader. State: Replica: 70ef05fecf19496cb1e4fc6ccb7fe6c2, State: Running, Role: LEADER
I20251026 21:06:35.795226 7454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } }
I20251026 21:06:35.795991 7449 sys_catalog.cc:565] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:35.796659 7454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ef05fecf19496cb1e4fc6ccb7fe6c2. Latest consensus state: current_term: 1 leader_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } } }
I20251026 21:06:35.796774 7454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:35.797185 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ef05fecf19496cb1e4fc6ccb7fe6c2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 42097 } } }
I20251026 21:06:35.797343 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:35.798045 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:35.784681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.3.175.62:42097)
1026 21:06:35.785096 (+ 415us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:35.785105 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:35.785141 (+ 36us) server_negotiation.cc:408] Connection header received
1026 21:06:35.785744 (+ 603us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:35.785763 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:35.786102 (+ 339us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:35.786447 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:35.787475 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:35.788271 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:35.789110 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:35.789314 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:35.791907 (+ 2593us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:35.791937 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:35.791953 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:35.792013 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:35.793986 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:35.794544 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:35.794551 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:35.794557 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:35.794743 (+ 186us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:35.795109 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:35.795112 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:35.795114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:35.795604 (+ 490us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:35.796747 (+ 1143us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:35.796954 (+ 207us) server_negotiation.cc:300] Negotiation successful
1026 21:06:35.797240 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
W20251026 21:06:35.801363 7469 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 21:06:35.801436 7469 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 21:06:35.801487 7459 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:35.802421 7459 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:35.807798 7459 catalog_manager.cc:1357] Generated new cluster ID: 9771efb42eee46239c78e7c208c51665
I20251026 21:06:35.807878 7459 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:35.819250 7459 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:35.820451 7459 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:35.828775 7459 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 70ef05fecf19496cb1e4fc6ccb7fe6c2: Generated new TSK 0
I20251026 21:06:35.829567 7459 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:35.886677 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:42097
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:36.003890 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:36.004177 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:36.004276 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:36.007838 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:36.007944 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:36.008069 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:36.012780 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:36.014021 7477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:36.014945 7477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:36.022140 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:36.022147 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:36.022153 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:36.022485 7477 server_base.cc:1047] running on GCE node
I20251026 21:06:36.022897 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:36.023553 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:36.024705 7477 hybrid_clock.cc:648] HybridClock initialized: now 1761512796024692 us; error 39 us; skew 500 ppm
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:36.027632 7477 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:36.028750 7477 webserver.cc:492] Webserver started at http://127.3.175.1:44309/ using document root <none> and password file <none>
I20251026 21:06:36.029332 7477 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:36.029387 7477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:36.029714 7477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:36.031697 7477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "32577bee989d4e5491d3170231d8a775"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ff53343585a38c3ae4d886150cb8d9c3"
server_key_iv: "0c6a0ce89067fd5d8686c06aab1a2d29"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.032225 7477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "32577bee989d4e5491d3170231d8a775"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ff53343585a38c3ae4d886150cb8d9c3"
server_key_iv: "0c6a0ce89067fd5d8686c06aab1a2d29"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.036087 7477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251026 21:06:36.038594 7492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.039806 7477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:36.039942 7477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "32577bee989d4e5491d3170231d8a775"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ff53343585a38c3ae4d886150cb8d9c3"
server_key_iv: "0c6a0ce89067fd5d8686c06aab1a2d29"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.040062 7477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:36.075619 7477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:36.078428 7477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:36.078616 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:36.079201 7477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:36.080123 7477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:36.080176 7477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.080240 7477 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:36.080268 7477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.089375 7477 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:41533
I20251026 21:06:36.089401 7605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:41533 every 8 connection(s)
I20251026 21:06:36.090387 7477 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:36.093732 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7477
I20251026 21:06:36.093873 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:36.094972 3772 external_mini_cluster.cc:1468] Setting key d5791e1faf89a610cef2ac3f2692f3e9
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:06:36.104277 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.092262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:34203 (local address 127.3.175.62:42097)
1026 21:06:36.092435 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:36.092443 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:36.093324 (+ 881us) server_negotiation.cc:408] Connection header received
1026 21:06:36.094528 (+ 1204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:36.094532 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:36.094604 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:36.094705 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:36.096388 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.096909 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.097595 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.097788 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.100627 (+ 2839us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:36.100652 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:36.100655 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:36.100685 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:36.102154 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.102751 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.102756 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.102757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.102809 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.103084 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.103087 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.103089 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.103254 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:36.103394 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:36.103946 (+ 552us) server_negotiation.cc:300] Negotiation successful
1026 21:06:36.104102 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:06:36.104962 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.092600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:42097 (local address 127.3.175.1:34203)
1026 21:06:36.093146 (+ 546us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:36.093196 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:36.094277 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:36.094904 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:36.094918 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:36.095603 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:36.096221 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.096236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.097044 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.097048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:36.097471 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.097478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.097726 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.098907 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:36.098935 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:36.100457 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:36.102312 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.102323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.102337 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.102653 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.102896 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.102899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.102901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.102999 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.103401 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:36.103410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:36.103694 (+ 284us) client_negotiation.cc:769] Sending connection context
1026 21:06:36.103904 (+ 210us) client_negotiation.cc:240] Negotiation successful
1026 21:06:36.104197 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:36.106123 7606 heartbeater.cc:344] Connected to a master server at 127.3.175.62:42097
I20251026 21:06:36.106367 7606 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:36.106907 7606 heartbeater.cc:507] Master 127.3.175.62:42097 requested a full tablet report, sending...
I20251026 21:06:36.108421 7413 ts_manager.cc:194] Registered new tserver with Master: 32577bee989d4e5491d3170231d8a775 (127.3.175.1:41533)
I20251026 21:06:36.109572 7413 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:34203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:36.146284 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:42097
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:36.257745 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:36.258013 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:36.258073 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:36.261525 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:36.261597 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:36.261682 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:36.266177 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:36.267452 7613 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:36.268321 7613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:36.274873 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:36.274863 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:36.275070 7613 server_base.cc:1047] running on GCE node
W20251026 21:06:36.274881 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:36.275534 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:36.276125 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:36.277307 7613 hybrid_clock.cc:648] HybridClock initialized: now 1761512796277291 us; error 29 us; skew 500 ppm
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:36.280356 7613 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:36.281487 7613 webserver.cc:492] Webserver started at http://127.3.175.2:43353/ using document root <none> and password file <none>
I20251026 21:06:36.282095 7613 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:36.282152 7613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:36.282341 7613 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:36.284190 7613 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "c057e84106be45dd962c94800ae7ade0"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ab0c88be789fcb1ed872a23ac65b9ec4"
server_key_iv: "a51c57314e0febdab27f5db54f71264b"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.284689 7613 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "c057e84106be45dd962c94800ae7ade0"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ab0c88be789fcb1ed872a23ac65b9ec4"
server_key_iv: "a51c57314e0febdab27f5db54f71264b"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.288371 7613 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251026 21:06:36.290936 7628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.292075 7613 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:36.292197 7613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "c057e84106be45dd962c94800ae7ade0"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ab0c88be789fcb1ed872a23ac65b9ec4"
server_key_iv: "a51c57314e0febdab27f5db54f71264b"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.292297 7613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:36.313930 7613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:36.317029 7613 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:36.317216 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:36.317764 7613 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:36.318634 7613 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:36.318683 7613 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.318740 7613 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:36.318764 7613 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.327651 7613 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:39485
I20251026 21:06:36.327677 7741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:39485 every 8 connection(s)
I20251026 21:06:36.328737 7613 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:36.332621 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7613
I20251026 21:06:36.332720 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:36.332968 3772 external_mini_cluster.cc:1468] Setting key 8126a29452b5e134f2588810ec71b4ee
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:36.342535 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.330566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:41289 (local address 127.3.175.62:42097)
1026 21:06:36.330691 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:36.330713 (+ 22us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:36.331445 (+ 732us) server_negotiation.cc:408] Connection header received
1026 21:06:36.332269 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:36.332273 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:36.332326 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:36.332400 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:36.333744 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.334428 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.335332 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.335468 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.338294 (+ 2826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:36.338322 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:36.338329 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:36.338367 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:36.340322 (+ 1955us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.340889 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.340895 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.340897 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.340956 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.341300 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.341306 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.341311 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.341539 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:36.341669 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:36.342175 (+ 506us) server_negotiation.cc:300] Negotiation successful
1026 21:06:36.342325 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 21:06:36.343143 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.330892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:42097 (local address 127.3.175.2:41289)
1026 21:06:36.331320 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:36.331356 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:36.332071 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:36.332531 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:36.332540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:36.333008 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:36.333617 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.333631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.334554 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.334561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:36.335184 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.335198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.335482 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.336634 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:36.336660 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:36.338125 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:36.340507 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.340512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.340522 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.340789 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.341064 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.341066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.341067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.341201 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.341656 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:36.341662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:36.341916 (+ 254us) client_negotiation.cc:769] Sending connection context
1026 21:06:36.342101 (+ 185us) client_negotiation.cc:240] Negotiation successful
1026 21:06:36.342365 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20251026 21:06:36.344321 7742 heartbeater.cc:344] Connected to a master server at 127.3.175.62:42097
I20251026 21:06:36.344585 7742 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:36.345227 7742 heartbeater.cc:507] Master 127.3.175.62:42097 requested a full tablet report, sending...
I20251026 21:06:36.346271 7413 ts_manager.cc:194] Registered new tserver with Master: c057e84106be45dd962c94800ae7ade0 (127.3.175.2:39485)
I20251026 21:06:36.346841 7413 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:41289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:36.379758 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:42097
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:36.490602 7749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:36.490881 7749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:36.490943 7749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:36.494311 7749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:36.494395 7749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:36.494478 7749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:36.499107 7749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:36.500298 7749 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:36.501140 7749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 21:06:36.508241 7749 server_base.cc:1047] running on GCE node
W20251026 21:06:36.508155 7757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:36.508172 7755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:36.508157 7754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:36.508786 7749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:36.509371 7749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:36.510519 7749 hybrid_clock.cc:648] HybridClock initialized: now 1761512796510507 us; error 32 us; skew 500 ppm
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:36.513418 7749 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:36.514526 7749 webserver.cc:492] Webserver started at http://127.3.175.3:42313/ using document root <none> and password file <none>
I20251026 21:06:36.515125 7749 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:36.515178 7749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:36.515396 7749 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:36.517160 7749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "08cb8afcfeb54e7db037588a3f4c8c66"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ed1469e8f451d0b58f900045f8178b83"
server_key_iv: "ddfbf78085798ea647e58028bab5311c"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.517659 7749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "08cb8afcfeb54e7db037588a3f4c8c66"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ed1469e8f451d0b58f900045f8178b83"
server_key_iv: "ddfbf78085798ea647e58028bab5311c"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.521283 7749 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:36.523645 7764 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.524742 7749 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:06:36.524859 7749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "08cb8afcfeb54e7db037588a3f4c8c66"
format_stamp: "Formatted at 2025-10-26 21:06:36 on dist-test-slave-th6x"
server_key: "ed1469e8f451d0b58f900045f8178b83"
server_key_iv: "ddfbf78085798ea647e58028bab5311c"
server_key_version: "encryptionkey@0"
I20251026 21:06:36.524957 7749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:36.538532 7749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:36.541543 7749 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:36.541713 7749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:36.542228 7749 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:36.543112 7749 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:36.543159 7749 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.543201 7749 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:36.543222 7749 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:36.552176 7749 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:38089
I20251026 21:06:36.552206 7877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:38089 every 8 connection(s)
I20251026 21:06:36.553185 7749 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:36.555815 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7749
I20251026 21:06:36.555948 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:36.556229 3772 external_mini_cluster.cc:1468] Setting key c73e43c2de7bfa9fa5ba2a6fd23da1a9
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:36.565402 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.554935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:48865 (local address 127.3.175.62:42097)
1026 21:06:36.555083 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:36.555087 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:36.555827 (+ 740us) server_negotiation.cc:408] Connection header received
1026 21:06:36.556780 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:36.556784 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:36.556839 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:36.556925 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:36.558448 (+ 1523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.558948 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.559596 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.559762 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.562326 (+ 2564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:36.562343 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:36.562346 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:36.562375 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:36.563780 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.564223 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.564225 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.564227 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.564268 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.564469 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.564471 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.564472 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.564628 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:36.564695 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:36.565176 (+ 481us) server_negotiation.cc:300] Negotiation successful
1026 21:06:36.565273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:06:36.566115 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.555227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:42097 (local address 127.3.175.3:48865)
1026 21:06:36.555688 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:36.555725 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:36.556568 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:36.557089 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:36.557103 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:36.557664 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:36.558319 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.558334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.559041 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.559043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:36.559493 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:36.559501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.559671 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.560722 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:36.560746 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:36.562171 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:36.563877 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.563881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.563891 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.564136 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.564329 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:36.564331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:36.564332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:36.564407 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:36.564703 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:36.564708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:36.564954 (+ 246us) client_negotiation.cc:769] Sending connection context
1026 21:06:36.565152 (+ 198us) client_negotiation.cc:240] Negotiation successful
1026 21:06:36.565376 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20251026 21:06:36.567217 7878 heartbeater.cc:344] Connected to a master server at 127.3.175.62:42097
I20251026 21:06:36.567492 7878 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:36.567999 7878 heartbeater.cc:507] Master 127.3.175.62:42097 requested a full tablet report, sending...
I20251026 21:06:36.569064 7413 ts_manager.cc:194] Registered new tserver with Master: 08cb8afcfeb54e7db037588a3f4c8c66 (127.3.175.3:38089)
I20251026 21:06:36.569741 7413 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:48865
I20251026 21:06:36.569831 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512796, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:36.595379 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:36.587922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.3.175.62:42097)
1026 21:06:36.588064 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:36.588068 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:36.588282 (+ 214us) server_negotiation.cc:408] Connection header received
1026 21:06:36.588398 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:36.588401 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:36.588452 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:36.588521 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:36.589359 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.589841 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:36.590394 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:36.590515 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:36.592762 (+ 2247us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:36.592785 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:36.592788 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:36.592829 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:36.594094 (+ 1265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.594452 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.594454 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.594456 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.594491 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:36.594636 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:36.594638 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:36.594639 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:36.594818 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:36.594917 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:36.595074 (+ 157us) server_negotiation.cc:300] Negotiation successful
1026 21:06:36.595170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251026 21:06:36.595813 7413 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51602
I20251026 21:06:36.596596 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7477
I20251026 21:06:36.602526 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7613
I20251026 21:06:36.608767 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7749
I20251026 21:06:36.614182 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7381
2025-10-26T21:06:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3169 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7890](info): set up 2 sockets
Oct 26 21:06:36 dist-test-slave-th6x krb5kdc[7890](info): commencing operation
krb5kdc: starting...
W20251026 21:06:38.615633 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:38 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:38.754406 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46691
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:38.864588 7906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:38.864854 7906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:38.864912 7906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:38.868265 7906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:38.868340 7906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:38.868364 7906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:38.868383 7906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:38.868402 7906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:38.872901 7906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:38.874078 7906 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:38.874941 7906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:38.880791 7914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:38.880782 7912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:38.880782 7911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:38.880997 7906 server_base.cc:1047] running on GCE node
I20251026 21:06:38.881647 7906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:38.882656 7906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:38.883948 7906 hybrid_clock.cc:648] HybridClock initialized: now 1761512798883936 us; error 30 us; skew 500 ppm
Oct 26 21:06:38 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:38.886507 7906 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:38.887655 7906 webserver.cc:492] Webserver started at http://127.3.175.62:43367/ using document root <none> and password file <none>
I20251026 21:06:38.888222 7906 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:38.888276 7906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:38.888465 7906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:38.890277 7906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "390b03b984cf4e4c944b36dfc14ae838"
format_stamp: "Formatted at 2025-10-26 21:06:38 on dist-test-slave-th6x"
server_key: "64ccfde4265974495e31e31b41e3d8fd"
server_key_iv: "c7a6baacbbb128fd898d966f8600d99a"
server_key_version: "encryptionkey@0"
I20251026 21:06:38.890769 7906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "390b03b984cf4e4c944b36dfc14ae838"
format_stamp: "Formatted at 2025-10-26 21:06:38 on dist-test-slave-th6x"
server_key: "64ccfde4265974495e31e31b41e3d8fd"
server_key_iv: "c7a6baacbbb128fd898d966f8600d99a"
server_key_version: "encryptionkey@0"
I20251026 21:06:38.894284 7906 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:38.896607 7921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:38.897681 7906 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:06:38.897823 7906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "390b03b984cf4e4c944b36dfc14ae838"
format_stamp: "Formatted at 2025-10-26 21:06:38 on dist-test-slave-th6x"
server_key: "64ccfde4265974495e31e31b41e3d8fd"
server_key_iv: "c7a6baacbbb128fd898d966f8600d99a"
server_key_version: "encryptionkey@0"
I20251026 21:06:38.897939 7906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:38.929967 7906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:38.936833 7906 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:38.937045 7906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:38.944664 7973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:46691 every 8 connection(s)
I20251026 21:06:38.944669 7906 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:46691
I20251026 21:06:38.945793 7906 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:38.948750 7974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:38.950555 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 7906
I20251026 21:06:38.950824 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:38.951083 3772 external_mini_cluster.cc:1468] Setting key 4ee6d7ce0c735e63741bc9316bc9f2d7
I20251026 21:06:38.955592 7974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838: Bootstrap starting.
I20251026 21:06:38.958325 7974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838: Neither blocks nor log segments found. Creating new log.
Oct 26 21:06:38 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:38.959362 7974 log.cc:826] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:38.961818 7974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838: No bootstrap required, opened a new log
I20251026 21:06:38.964646 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:38.952521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.3.175.62:46691)
1026 21:06:38.952936 (+ 415us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:38.952947 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:38.952984 (+ 37us) server_negotiation.cc:408] Connection header received
1026 21:06:38.953528 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:38.953546 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:38.953863 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:38.954154 (+ 291us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:38.955343 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:38.956249 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:38.956930 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:38.957271 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:38.960001 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:38.960025 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:38.960042 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:38.960083 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:38.962110 (+ 2027us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:38.962601 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:38.962605 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:38.962610 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:38.962677 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:38.962902 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:38.962904 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:38.962906 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:38.963255 (+ 349us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:38.963450 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:38.963734 (+ 284us) server_negotiation.cc:300] Negotiation successful
1026 21:06:38.963991 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
I20251026 21:06:38.964973 7974 raft_consensus.cc:359] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } }
I20251026 21:06:38.965173 7974 raft_consensus.cc:385] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:38.965229 7974 raft_consensus.cc:740] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b03b984cf4e4c944b36dfc14ae838, State: Initialized, Role: FOLLOWER
I20251026 21:06:38.965759 7974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } }
I20251026 21:06:38.965910 7974 raft_consensus.cc:399] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:38.965988 7974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:38.966084 7974 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:38.967018 7974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } }
I20251026 21:06:38.967382 7974 leader_election.cc:304] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 390b03b984cf4e4c944b36dfc14ae838; no voters:
I20251026 21:06:38.967675 7974 leader_election.cc:290] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:38.967820 7979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:38.968092 7979 raft_consensus.cc:697] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [term 1 LEADER]: Becoming Leader. State: Replica: 390b03b984cf4e4c944b36dfc14ae838, State: Running, Role: LEADER
I20251026 21:06:38.968400 7979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } }
I20251026 21:06:38.968832 7974 sys_catalog.cc:565] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:38.969522 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 390b03b984cf4e4c944b36dfc14ae838. Latest consensus state: current_term: 1 leader_uuid: "390b03b984cf4e4c944b36dfc14ae838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } } }
I20251026 21:06:38.969642 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:38.969944 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "390b03b984cf4e4c944b36dfc14ae838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390b03b984cf4e4c944b36dfc14ae838" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46691 } } }
I20251026 21:06:38.970118 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:38.970363 7984 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:38.973006 7984 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:38.978540 7984 catalog_manager.cc:1357] Generated new cluster ID: eb64a18fe38e4761831b9cf894631c42
I20251026 21:06:38.978616 7984 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:39.005626 7984 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:39.006609 7984 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:39.012974 7984 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 390b03b984cf4e4c944b36dfc14ae838: Generated new TSK 0
I20251026 21:06:39.013814 7984 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:39.072319 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:39.184576 8002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:39.184836 8002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:39.184908 8002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:39.188177 8002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:39.188253 8002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:39.188375 8002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:39.192685 8002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:39.193840 8002 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:39.194712 8002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:39.201395 8010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:39.201424 8007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:39.201654 8008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:39.201644 8002 server_base.cc:1047] running on GCE node
I20251026 21:06:39.202065 8002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:39.202689 8002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:39.203881 8002 hybrid_clock.cc:648] HybridClock initialized: now 1761512799203870 us; error 29 us; skew 500 ppm
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:39.206871 8002 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:39.208169 8002 webserver.cc:492] Webserver started at http://127.3.175.1:38135/ using document root <none> and password file <none>
I20251026 21:06:39.208765 8002 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:39.208823 8002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:39.209142 8002 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:39.211028 8002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "d45c35966007444aa1bff2ec8828f872"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "ce33738c5a91c2e9f0212163208141f8"
server_key_iv: "0e11f27de768aa44feaa6a70336a469f"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.211571 8002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "d45c35966007444aa1bff2ec8828f872"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "ce33738c5a91c2e9f0212163208141f8"
server_key_iv: "0e11f27de768aa44feaa6a70336a469f"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.215206 8002 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251026 21:06:39.217515 8017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.218556 8002 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:39.218648 8002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "d45c35966007444aa1bff2ec8828f872"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "ce33738c5a91c2e9f0212163208141f8"
server_key_iv: "0e11f27de768aa44feaa6a70336a469f"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.218726 8002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:39.231950 8002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:39.234956 8002 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:39.235129 8002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:39.235718 8002 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:39.236608 8002 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:39.236661 8002 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.236730 8002 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:39.236763 8002 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.245774 8002 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:40361
I20251026 21:06:39.245801 8130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:40361 every 8 connection(s)
I20251026 21:06:39.246768 8002 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:39.248859 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8002
I20251026 21:06:39.248976 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:39.249293 3772 external_mini_cluster.cc:1468] Setting key e41959a670bbe8c3da0b0b490aab6bd2
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:06:39.260485 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.248556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:56513 (local address 127.3.175.62:46691)
1026 21:06:39.248730 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:39.248736 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:39.249518 (+ 782us) server_negotiation.cc:408] Connection header received
1026 21:06:39.250448 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:39.250452 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:39.250498 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:39.250593 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:39.252855 (+ 2262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.253689 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.254332 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.254571 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.256926 (+ 2355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:39.256951 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:39.256956 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:39.256990 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:39.258663 (+ 1673us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.259140 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.259143 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.259144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.259188 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.259459 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.259461 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.259463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.259614 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:39.259691 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:39.260243 (+ 552us) server_negotiation.cc:300] Negotiation successful
1026 21:06:39.260357 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 21:06:39.261193 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.248847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46691 (local address 127.3.175.1:56513)
1026 21:06:39.249366 (+ 519us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:39.249413 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:39.250258 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:39.250740 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:39.250752 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:39.251381 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:39.252697 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.252713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.253809 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.253811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:39.254219 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.254225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.254414 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.255034 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:39.255059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:39.256759 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:39.258800 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.258808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.258821 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.259049 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.259260 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.259262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.259263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.259393 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.259703 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:39.259708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:39.259984 (+ 276us) client_negotiation.cc:769] Sending connection context
1026 21:06:39.260196 (+ 212us) client_negotiation.cc:240] Negotiation successful
1026 21:06:39.260422 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:39.262409 8131 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46691
I20251026 21:06:39.262646 8131 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:39.263161 8131 heartbeater.cc:507] Master 127.3.175.62:46691 requested a full tablet report, sending...
I20251026 21:06:39.264600 7938 ts_manager.cc:194] Registered new tserver with Master: d45c35966007444aa1bff2ec8828f872 (127.3.175.1:40361)
I20251026 21:06:39.265764 7938 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:56513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:39.301924 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:39.410620 8138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:39.410878 8138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:39.410964 8138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:39.414500 8138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:39.414580 8138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:39.414693 8138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:39.419253 8138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:39.420480 8138 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:39.421361 8138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:39.428516 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:39.428517 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:39.428517 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:39.428663 8138 server_base.cc:1047] running on GCE node
I20251026 21:06:39.429355 8138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:39.429921 8138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:39.431075 8138 hybrid_clock.cc:648] HybridClock initialized: now 1761512799431063 us; error 31 us; skew 500 ppm
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:39.433722 8138 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:39.434814 8138 webserver.cc:492] Webserver started at http://127.3.175.2:45071/ using document root <none> and password file <none>
I20251026 21:06:39.435408 8138 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:39.435464 8138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:39.435653 8138 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:39.437420 8138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "48e0297ea8ad48f8a36ca5974c8416f2"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "1505fabba7bc46c15f026535b1e4211a"
server_key_iv: "98b0229f21f48e0a663cf2c5b8188815"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.437886 8138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "48e0297ea8ad48f8a36ca5974c8416f2"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "1505fabba7bc46c15f026535b1e4211a"
server_key_iv: "98b0229f21f48e0a663cf2c5b8188815"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.441399 8138 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:39.443636 8153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.444658 8138 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:39.444769 8138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "48e0297ea8ad48f8a36ca5974c8416f2"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "1505fabba7bc46c15f026535b1e4211a"
server_key_iv: "98b0229f21f48e0a663cf2c5b8188815"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.444860 8138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:39.458438 8138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:39.461611 8138 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:39.461814 8138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:39.462420 8138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:39.463402 8138 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:39.463451 8138 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.463511 8138 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:39.463539 8138 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.472779 8138 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:35683
I20251026 21:06:39.472812 8266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:35683 every 8 connection(s)
I20251026 21:06:39.473807 8138 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:39.478087 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8138
I20251026 21:06:39.478207 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:39.478474 3772 external_mini_cluster.cc:1468] Setting key 3f2fd0918d966ceb75284f1f9bce0b30
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:39.487404 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.475632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:45877 (local address 127.3.175.62:46691)
1026 21:06:39.475770 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:39.475774 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:39.476503 (+ 729us) server_negotiation.cc:408] Connection header received
1026 21:06:39.477374 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:39.477377 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:39.477442 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:39.477518 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:39.479612 (+ 2094us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.480133 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.480788 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.480980 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.483498 (+ 2518us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:39.483520 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:39.483522 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:39.483561 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:39.485087 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.485763 (+ 676us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.485766 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.485767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.485806 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.486120 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.486122 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.486123 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.486272 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:39.486343 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:39.487052 (+ 709us) server_negotiation.cc:300] Negotiation successful
1026 21:06:39.487209 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:06:39.488229 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.475919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46691 (local address 127.3.175.2:45877)
1026 21:06:39.476369 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:39.476406 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:39.477171 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:39.477692 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:39.477701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:39.478291 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:39.479457 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.479474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.480260 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.480263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:39.480668 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.480675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.480929 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.481570 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:39.481597 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:39.483313 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:39.485234 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.485242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.485258 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.485645 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.485885 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.485889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.485891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.486038 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.486365 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:39.486375 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:39.486750 (+ 375us) client_negotiation.cc:769] Sending connection context
1026 21:06:39.487007 (+ 257us) client_negotiation.cc:240] Negotiation successful
1026 21:06:39.487520 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20251026 21:06:39.489352 8267 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46691
I20251026 21:06:39.489603 8267 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:39.490176 8267 heartbeater.cc:507] Master 127.3.175.62:46691 requested a full tablet report, sending...
I20251026 21:06:39.491266 7938 ts_manager.cc:194] Registered new tserver with Master: 48e0297ea8ad48f8a36ca5974c8416f2 (127.3.175.2:35683)
I20251026 21:06:39.491902 7938 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:45877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:39.526495 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:39.639988 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:39.640244 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:39.640342 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:39.643762 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:39.643838 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:39.643950 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:39.648350 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:39.649567 8274 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:39.650461 8274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:39.657290 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:39.657418 8274 server_base.cc:1047] running on GCE node
W20251026 21:06:39.657291 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:39.657291 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:39.657895 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:39.658439 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:39.659597 8274 hybrid_clock.cc:648] HybridClock initialized: now 1761512799659584 us; error 32 us; skew 500 ppm
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:39.662465 8274 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:39.663626 8274 webserver.cc:492] Webserver started at http://127.3.175.3:37095/ using document root <none> and password file <none>
I20251026 21:06:39.664214 8274 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:39.664270 8274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:39.664454 8274 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:39.666312 8274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "f962597b312248b99f6899d7ffe91e03"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "0dec3da0f74c5e3f1f4a02b58477d956"
server_key_iv: "06346054ecc3bda84bda335e7ac0d2a0"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.666802 8274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "f962597b312248b99f6899d7ffe91e03"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "0dec3da0f74c5e3f1f4a02b58477d956"
server_key_iv: "06346054ecc3bda84bda335e7ac0d2a0"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.670403 8274 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:06:39.672760 8289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.673820 8274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251026 21:06:39.673928 8274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "f962597b312248b99f6899d7ffe91e03"
format_stamp: "Formatted at 2025-10-26 21:06:39 on dist-test-slave-th6x"
server_key: "0dec3da0f74c5e3f1f4a02b58477d956"
server_key_iv: "06346054ecc3bda84bda335e7ac0d2a0"
server_key_version: "encryptionkey@0"
I20251026 21:06:39.674022 8274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:39.699083 8274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:39.702163 8274 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:39.702350 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:39.702955 8274 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:39.703883 8274 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:39.703934 8274 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.703992 8274 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:39.704031 8274 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:39.713445 8274 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:34877
I20251026 21:06:39.713472 8402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:34877 every 8 connection(s)
I20251026 21:06:39.714428 8274 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:39.723007 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8274
I20251026 21:06:39.723160 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:39.723431 3772 external_mini_cluster.cc:1468] Setting key 27c6178add6674153560289fae5df37c
I20251026 21:06:39.726866 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.716247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:50871 (local address 127.3.175.62:46691)
1026 21:06:39.716380 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:39.716384 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:39.717081 (+ 697us) server_negotiation.cc:408] Connection header received
1026 21:06:39.717971 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:39.717975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:39.718053 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:39.718129 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:39.719865 (+ 1736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.720376 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.721016 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.721200 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.723562 (+ 2362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:39.723585 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:39.723587 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:39.723615 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:39.725175 (+ 1560us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.725686 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.725689 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.725691 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.725739 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.725970 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.725972 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.725974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.726124 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:39.726193 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:39.726618 (+ 425us) server_negotiation.cc:300] Negotiation successful
1026 21:06:39.726714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:06:39.727628 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.716487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46691 (local address 127.3.175.3:50871)
1026 21:06:39.716932 (+ 445us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:39.716970 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:39.717727 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:39.718299 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:39.718308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:39.718706 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:39.719707 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.719721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.720500 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.720503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:39.720910 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:39.720916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.721129 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.721843 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:39.721867 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:39.723382 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:39.725288 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.725293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.725302 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.725603 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.725817 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:39.725819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:39.725821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:39.725904 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:39.726189 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:39.726194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:39.726415 (+ 221us) client_negotiation.cc:769] Sending connection context
1026 21:06:39.726609 (+ 194us) client_negotiation.cc:240] Negotiation successful
1026 21:06:39.726847 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20251026 21:06:39.728668 8403 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46691
I20251026 21:06:39.728894 8403 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:39.729379 8403 heartbeater.cc:507] Master 127.3.175.62:46691 requested a full tablet report, sending...
I20251026 21:06:39.730297 7938 ts_manager.cc:194] Registered new tserver with Master: f962597b312248b99f6899d7ffe91e03 (127.3.175.3:34877)
I20251026 21:06:39.730841 7938 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:50871
I20251026 21:06:39.737428 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:06:39.761273 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.752798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.3.175.1:40361)
1026 21:06:39.753107 (+ 309us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:39.753113 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:39.753140 (+ 27us) server_negotiation.cc:408] Connection header received
1026 21:06:39.753196 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:39.753200 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:39.753373 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:39.753499 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:39.754299 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.755006 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.755674 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.755883 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.758041 (+ 2158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:39.758071 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:39.758084 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:39.758114 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:39.759833 (+ 1719us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.760243 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.760248 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.760250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.760300 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.760467 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.760469 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.760470 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.760659 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:39.760821 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:39.760936 (+ 115us) server_negotiation.cc:300] Negotiation successful
1026 21:06:39.761063 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:39.771052 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:39.764007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.3.175.62:46691)
1026 21:06:39.764142 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:39.764146 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:39.764194 (+ 48us) server_negotiation.cc:408] Connection header received
1026 21:06:39.764304 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:39.764307 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:39.764353 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:39.764409 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:39.765073 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.765562 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:39.766087 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:39.766208 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:39.768392 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:39.768413 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:39.768415 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:39.768444 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:39.769713 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.770078 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.770081 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.770082 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.770117 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:39.770275 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:39.770277 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:39.770278 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:39.770474 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:39.770575 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:39.770734 (+ 159us) server_negotiation.cc:300] Negotiation successful
1026 21:06:39.770866 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20251026 21:06:39.771550 7938 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57758
I20251026 21:06:39.772161 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8002
I20251026 21:06:39.778386 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8138
I20251026 21:06:39.786450 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8274
I20251026 21:06:39.792107 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 7906
2025-10-26T21:06:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3177 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[8415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[8415](info): set up 2 sockets
Oct 26 21:06:39 dist-test-slave-th6x krb5kdc[8415](info): commencing operation
krb5kdc: starting...
W20251026 21:06:41.803838 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:06:41 dist-test-slave-th6x krb5kdc[8415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:06:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:41.946235 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34835
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:34835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:42.058363 8431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:42.058650 8431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:42.058754 8431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:42.062170 8431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:42.062259 8431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:42.062299 8431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:42.062331 8431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:42.062369 8431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:42.066987 8431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:34835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:42.068269 8431 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:42.069187 8431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:42.075084 8439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.075078 8436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.075106 8437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.075253 8431 server_base.cc:1047] running on GCE node
I20251026 21:06:42.075927 8431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:42.076921 8431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:42.078074 8431 hybrid_clock.cc:648] HybridClock initialized: now 1761512802078063 us; error 32 us; skew 500 ppm
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:42.080902 8431 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:06:42.081708 8431 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:42.081775 8431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:42.082017 8431 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:42.083935 8431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "3a3df75443c0494699d9a6ebcaddb181"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1341e39fe5db34363f86c6909fe8494f"
server_key_iv: "2e5043ef74e908111c422ccecaff0c79"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.084460 8431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "3a3df75443c0494699d9a6ebcaddb181"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1341e39fe5db34363f86c6909fe8494f"
server_key_iv: "2e5043ef74e908111c422ccecaff0c79"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.088183 8431 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:42.090595 8445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.091843 8431 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:42.091970 8431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "3a3df75443c0494699d9a6ebcaddb181"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1341e39fe5db34363f86c6909fe8494f"
server_key_iv: "2e5043ef74e908111c422ccecaff0c79"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.092087 8431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:42.110010 8431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:42.113035 8431 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:42.113245 8431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:42.121549 8431 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:34835
I20251026 21:06:42.121526 8497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:34835 every 8 connection(s)
I20251026 21:06:42.122234 8431 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:42.125069 8498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:42.130753 8498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: Bootstrap starting.
I20251026 21:06:42.132354 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8431
I20251026 21:06:42.132498 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:42.132721 3772 external_mini_cluster.cc:1468] Setting key 396bc9b5cff11e1c15acecbab5c26365
I20251026 21:06:42.133270 8498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:42.134142 8498 log.cc:826] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:42.136286 8498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: No bootstrap required, opened a new log
I20251026 21:06:42.139467 8498 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } }
I20251026 21:06:42.139743 8498 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:42.139816 8498 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3df75443c0494699d9a6ebcaddb181, State: Initialized, Role: FOLLOWER
I20251026 21:06:42.140311 8498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } }
I20251026 21:06:42.140440 8498 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:42.140508 8498 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:42.140599 8498 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:42.141769 8498 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } }
I20251026 21:06:42.142212 8498 leader_election.cc:304] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a3df75443c0494699d9a6ebcaddb181; no voters:
I20251026 21:06:42.142561 8498 leader_election.cc:290] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:42.142678 8503 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:42.142920 8503 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [term 1 LEADER]: Becoming Leader. State: Replica: 3a3df75443c0494699d9a6ebcaddb181, State: Running, Role: LEADER
I20251026 21:06:42.143326 8503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } }
I20251026 21:06:42.144014 8498 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:42.144445 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a3df75443c0494699d9a6ebcaddb181. Latest consensus state: current_term: 1 leader_uuid: "3a3df75443c0494699d9a6ebcaddb181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } } }
I20251026 21:06:42.144657 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:42.144852 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a3df75443c0494699d9a6ebcaddb181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a3df75443c0494699d9a6ebcaddb181" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34835 } } }
I20251026 21:06:42.144937 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:42.145695 8507 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:42.147954 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.133723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.3.175.62:34835)
1026 21:06:42.134141 (+ 418us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.134150 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.134185 (+ 35us) server_negotiation.cc:408] Connection header received
1026 21:06:42.134928 (+ 743us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.134954 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.135247 (+ 293us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.135571 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:42.136678 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.137429 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.138293 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.138503 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.141160 (+ 2657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:42.141189 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:42.141204 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:42.141255 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:42.144164 (+ 2909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.144753 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.144759 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.144765 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.144875 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.145163 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.145166 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.145168 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.145625 (+ 457us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:42.146636 (+ 1011us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.146860 (+ 224us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.147146 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20251026 21:06:42.148880 8507 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:42.154595 8507 catalog_manager.cc:1357] Generated new cluster ID: 80bb659e84c74342bc05031094f864f5
I20251026 21:06:42.154852 8507 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:42.165138 8507 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:42.166307 8507 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:42.173056 8507 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: Generated new TSK 0
I20251026 21:06:42.173868 8507 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:42.228207 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34835
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:42.339663 8526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:42.339903 8526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:42.339978 8526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:42.343364 8526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:42.343431 8526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:42.343534 8526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:42.347934 8526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:42.349123 8526 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:42.350033 8526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:42.356593 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.356600 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.356848 8526 server_base.cc:1047] running on GCE node
W20251026 21:06:42.356606 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.357326 8526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:42.357900 8526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:42.359071 8526 hybrid_clock.cc:648] HybridClock initialized: now 1761512802359051 us; error 32 us; skew 500 ppm
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:42.361708 8526 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:06:42.362452 8526 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:42.362516 8526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:42.362706 8526 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:42.364579 8526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "aabefc4fb92844308c2b26b48a573692"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "df63d675055343a7256f239d82a9cebd"
server_key_iv: "d613544dab49a3e30a2cd7d8fb8413de"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.365165 8526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "aabefc4fb92844308c2b26b48a573692"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "df63d675055343a7256f239d82a9cebd"
server_key_iv: "d613544dab49a3e30a2cd7d8fb8413de"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.368705 8526 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:42.370842 8540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.371909 8526 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:42.372016 8526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "aabefc4fb92844308c2b26b48a573692"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "df63d675055343a7256f239d82a9cebd"
server_key_iv: "d613544dab49a3e30a2cd7d8fb8413de"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.372119 8526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:42.389822 8526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:42.393522 8526 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:42.393726 8526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:42.394366 8526 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:42.395454 8526 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:42.395504 8526 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.395555 8526 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:42.395574 8526 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.405717 8526 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:44651
I20251026 21:06:42.405746 8653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:44651 every 8 connection(s)
I20251026 21:06:42.406308 8526 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:42.407976 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8526
I20251026 21:06:42.408106 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:42.408380 3772 external_mini_cluster.cc:1468] Setting key f549fc5f2f79698d0f4509b7a883e497
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:06:42.420647 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.408096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:53629 (local address 127.3.175.62:34835)
1026 21:06:42.408230 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.408234 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.409148 (+ 914us) server_negotiation.cc:408] Connection header received
1026 21:06:42.410159 (+ 1011us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.410163 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.410215 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.410295 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:42.412432 (+ 2137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.412940 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.413897 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.414054 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.417123 (+ 3069us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:42.417145 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:42.417147 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:42.417175 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:42.418724 (+ 1549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.419225 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.419227 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.419229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.419270 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.419535 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.419537 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.419538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.419780 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:42.419865 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.420361 (+ 496us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.420512 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 21:06:42.421644 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.408453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34835 (local address 127.3.175.1:53629)
1026 21:06:42.409006 (+ 553us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.409051 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.409970 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.410480 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.410489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.410895 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:42.412262 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.412283 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.413063 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.413069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.413743 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.413755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.414000 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.414693 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:42.414735 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:42.416933 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:42.418861 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.418866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.418878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.419130 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.419361 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.419363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.419365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.419455 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.419874 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:42.419880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:42.420123 (+ 243us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.420337 (+ 214us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.420670 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20251026 21:06:42.422833 8654 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34835
I20251026 21:06:42.423112 8654 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:42.423669 8654 heartbeater.cc:507] Master 127.3.175.62:34835 requested a full tablet report, sending...
I20251026 21:06:42.425113 8462 ts_manager.cc:194] Registered new tserver with Master: aabefc4fb92844308c2b26b48a573692 (127.3.175.1:44651)
I20251026 21:06:42.426281 8462 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:53629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:42.458127 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34835
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:42.568004 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:42.568241 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:42.568305 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:42.571460 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:42.571519 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:42.571599 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:42.575901 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:42.577001 8661 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:42.577870 8661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:42.584421 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.584437 8669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.584421 8667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.584718 8661 server_base.cc:1047] running on GCE node
I20251026 21:06:42.585074 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:42.585623 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:42.586766 8661 hybrid_clock.cc:648] HybridClock initialized: now 1761512802586740 us; error 42 us; skew 500 ppm
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:42.589416 8661 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:06:42.590142 8661 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:42.590206 8661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:42.590392 8661 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:42.592175 8661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "589def50af714e74baa324d7683528c5"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1452ed099daeeea66fc3ce3a1f8cb065"
server_key_iv: "ed2c8794cdc7fa60624075ce04f3df37"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.592686 8661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "589def50af714e74baa324d7683528c5"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1452ed099daeeea66fc3ce3a1f8cb065"
server_key_iv: "ed2c8794cdc7fa60624075ce04f3df37"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.596067 8661 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:42.598456 8675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.599547 8661 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:42.599665 8661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "589def50af714e74baa324d7683528c5"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "1452ed099daeeea66fc3ce3a1f8cb065"
server_key_iv: "ed2c8794cdc7fa60624075ce04f3df37"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.599823 8661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:42.627960 8661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:42.631131 8661 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:42.631357 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:42.631937 8661 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:42.632829 8661 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:42.632879 8661 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.632937 8661 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:42.632970 8661 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.642123 8661 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:40465
I20251026 21:06:42.642148 8788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:40465 every 8 connection(s)
I20251026 21:06:42.642704 8661 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:42.644268 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8661
I20251026 21:06:42.644399 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:42.644645 3772 external_mini_cluster.cc:1468] Setting key 3e78c723b784c48c45e9e41035a69a4f
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:06:42.656596 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.645256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:58069 (local address 127.3.175.62:34835)
1026 21:06:42.645387 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.645391 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.645405 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:06:42.646397 (+ 992us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.646401 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.646460 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.646546 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:42.648665 (+ 2119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.649173 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.650132 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.650273 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.653114 (+ 2841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:42.653135 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:42.653137 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:42.653167 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:42.654726 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.655255 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.655258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.655259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.655335 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.655589 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.655591 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.655592 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.655750 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:42.655834 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.656360 (+ 526us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.656467 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:06:42.657392 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.644798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34835 (local address 127.3.175.2:58069)
1026 21:06:42.645291 (+ 493us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.645337 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.646195 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.646765 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.646775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.647186 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:42.648501 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.648523 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.649309 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.649316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.649993 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.650005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.650234 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.650905 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:42.650950 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:42.652920 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:42.654856 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.654861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.654872 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.655163 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.655422 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.655425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.655426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.655517 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.655840 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:42.655846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:42.656112 (+ 266us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.656326 (+ 214us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.656586 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20251026 21:06:42.658588 8789 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34835
I20251026 21:06:42.658869 8789 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:42.659418 8789 heartbeater.cc:507] Master 127.3.175.62:34835 requested a full tablet report, sending...
I20251026 21:06:42.660583 8462 ts_manager.cc:194] Registered new tserver with Master: 589def50af714e74baa324d7683528c5 (127.3.175.2:40465)
I20251026 21:06:42.661440 8462 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:58069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:06:42.695680 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34835
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:06:42.805433 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:42.805676 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:42.805745 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:42.809154 8796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:42.809229 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:42.809331 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:42.813644 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:42.814769 8796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:42.815654 8796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:42.822234 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:42.822235 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.822490 8796 server_base.cc:1047] running on GCE node
W20251026 21:06:42.822235 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:42.822886 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:42.823482 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:42.824662 8796 hybrid_clock.cc:648] HybridClock initialized: now 1761512802824622 us; error 54 us; skew 500 ppm
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:06:42.827474 8796 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:06:42.828168 8796 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:42.828222 8796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:42.828392 8796 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:42.830047 8796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "fcca9a6ad5a240e4b221eca426911810"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "cc391b03bb99bd375fa8f94331e8e417"
server_key_iv: "7d8c7573eb53ffebdc16992cd5024c62"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.830476 8796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "fcca9a6ad5a240e4b221eca426911810"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "cc391b03bb99bd375fa8f94331e8e417"
server_key_iv: "7d8c7573eb53ffebdc16992cd5024c62"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.833859 8796 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251026 21:06:42.835960 8810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.836953 8796 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:42.837047 8796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "fcca9a6ad5a240e4b221eca426911810"
format_stamp: "Formatted at 2025-10-26 21:06:42 on dist-test-slave-th6x"
server_key: "cc391b03bb99bd375fa8f94331e8e417"
server_key_iv: "7d8c7573eb53ffebdc16992cd5024c62"
server_key_version: "encryptionkey@0"
I20251026 21:06:42.837131 8796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:42.851349 8796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:42.854306 8796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:42.854473 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:42.855001 8796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:42.855935 8796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:42.855993 8796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.856035 8796 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:42.856056 8796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:42.865015 8796 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:40099
I20251026 21:06:42.865036 8923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:40099 every 8 connection(s)
I20251026 21:06:42.865614 8796 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:42.871631 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8796
I20251026 21:06:42.871762 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:42.872010 3772 external_mini_cluster.cc:1468] Setting key e613312991b3971d7582d3691bc2ce3d
Oct 26 21:06:42 dist-test-slave-th6x krb5kdc[8415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:06:42.878259 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.867368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:34709 (local address 127.3.175.62:34835)
1026 21:06:42.867525 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.867529 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.868207 (+ 678us) server_negotiation.cc:408] Connection header received
1026 21:06:42.869121 (+ 914us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.869125 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.869175 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.869254 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:42.871049 (+ 1795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.871707 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.872429 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.872601 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.874832 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:42.874847 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:42.874850 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:42.874876 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:42.876317 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.876862 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.876865 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.876866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.876911 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.877200 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.877203 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.877205 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.877363 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:42.877454 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.877960 (+ 506us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.878082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:06:42.878927 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.867646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34835 (local address 127.3.175.3:34709)
1026 21:06:42.868066 (+ 420us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.868100 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.868806 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.869415 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.869423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.869839 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:42.870872 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.870889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.871872 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.871875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.872314 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.872323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.872536 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.873202 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:06:42.873227 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:42.874675 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:42.876438 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.876444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.876457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.876706 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.876998 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:06:42.877001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:06:42.877003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:06:42.877111 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:06:42.877464 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:42.877473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:42.877718 (+ 245us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.877916 (+ 198us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.878220 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20251026 21:06:42.880028 8924 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34835
I20251026 21:06:42.880299 8924 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:42.880800 8924 heartbeater.cc:507] Master 127.3.175.62:34835 requested a full tablet report, sending...
I20251026 21:06:42.881817 8462 ts_manager.cc:194] Registered new tserver with Master: fcca9a6ad5a240e4b221eca426911810 (127.3.175.3:40099)
I20251026 21:06:42.882364 8462 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:34709
I20251026 21:06:42.885489 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:06:42.893915 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.887272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.3.175.62:34835)
1026 21:06:42.887431 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.887437 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.887583 (+ 146us) server_negotiation.cc:408] Connection header received
1026 21:06:42.887713 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.887716 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.887764 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.887847 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:42.888708 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.889203 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.889811 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.890028 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.891162 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:42.891180 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:42.891182 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:06:42.891206 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:42.892455 (+ 1249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.892845 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.892848 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.892849 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.892892 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:06:42.893105 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:06:42.893108 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:06:42.893110 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:06:42.893287 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:42.893417 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.893621 (+ 204us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.893737 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 21:06:42.896560 8462 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:42.898869 8462 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:42.911795 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.907726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:40465 (local address 127.0.0.1:53786)
1026 21:06:42.908555 (+ 829us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.908569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.908647 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.909104 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.909108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.909127 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:42.909381 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.909388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910338 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.910341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.911210 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.911219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.911375 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.911440 (+ 65us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.911533 (+ 93us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.911626 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":103,"threads_started":1}
I20251026 21:06:42.912362 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.908122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.3.175.2:40465)
1026 21:06:42.908500 (+ 378us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.908506 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.908731 (+ 225us) server_negotiation.cc:408] Connection header received
1026 21:06:42.908808 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.908812 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.908955 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.909094 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:42.909499 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910226 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.911381 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.911942 (+ 561us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.912076 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.912142 (+ 66us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.912228 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20251026 21:06:42.912761 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.907711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.3.175.1:44651)
1026 21:06:42.908342 (+ 631us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.908350 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.908373 (+ 23us) server_negotiation.cc:408] Connection header received
1026 21:06:42.908458 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.908467 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.908659 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.908785 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:42.909232 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910383 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.911872 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.912371 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.912470 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.912543 (+ 73us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.912630 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":94,"threads_started":1}
I20251026 21:06:42.911795 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.907543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44651 (local address 127.0.0.1:39668)
1026 21:06:42.908174 (+ 631us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.908189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.908326 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.908791 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.908796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.908826 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:42.909097 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.909104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910506 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.910509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.911343 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.911351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.911436 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.911447 (+ 11us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.911533 (+ 86us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.911626 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":111,"threads_started":1}
I20251026 21:06:42.913203 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.907164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.3.175.3:40099)
1026 21:06:42.907899 (+ 735us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:42.907906 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:42.908139 (+ 233us) server_negotiation.cc:408] Connection header received
1026 21:06:42.908656 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:42.908663 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:42.908849 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:42.909002 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:42.909574 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910638 (+ 1064us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.911838 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.912662 (+ 824us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.912736 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:42.912961 (+ 225us) server_negotiation.cc:300] Negotiation successful
1026 21:06:42.913061 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":165,"threads_started":1}
I20251026 21:06:42.913581 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:42.907030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:40099 (local address 127.0.0.1:45762)
1026 21:06:42.907979 (+ 949us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:42.908020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:42.908510 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:42.909171 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:42.909174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:42.909207 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:42.909406 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.909411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.910759 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:42.910763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:42.911670 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:42.911680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:42.912789 (+ 1109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:42.912803 (+ 14us) client_negotiation.cc:769] Sending connection context
1026 21:06:42.913388 (+ 585us) client_negotiation.cc:240] Negotiation successful
1026 21:06:42.913460 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"spinlock_wait_cycles":3968,"thread_start_us":85,"threads_started":1}
I20251026 21:06:42.915519 8723 tablet_service.cc:1505] Processing CreateTablet for tablet 1b378e421d3f47cb8056967af7694fc2 (DEFAULT_TABLE table=test-table [id=b1febb61a54d415694208aec83e8bead]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:42.915519 8588 tablet_service.cc:1505] Processing CreateTablet for tablet 1b378e421d3f47cb8056967af7694fc2 (DEFAULT_TABLE table=test-table [id=b1febb61a54d415694208aec83e8bead]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:42.915922 8858 tablet_service.cc:1505] Processing CreateTablet for tablet 1b378e421d3f47cb8056967af7694fc2 (DEFAULT_TABLE table=test-table [id=b1febb61a54d415694208aec83e8bead]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:42.916549 8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b378e421d3f47cb8056967af7694fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:42.916549 8588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b378e421d3f47cb8056967af7694fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:42.916723 8858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b378e421d3f47cb8056967af7694fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:42.922416 8941 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Bootstrap starting.
I20251026 21:06:42.923568 8943 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Bootstrap starting.
I20251026 21:06:42.923645 8942 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Bootstrap starting.
I20251026 21:06:42.925323 8943 tablet_bootstrap.cc:654] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:42.925550 8942 tablet_bootstrap.cc:654] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:42.925550 8941 tablet_bootstrap.cc:654] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:42.926092 8943 log.cc:826] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:42.926363 8941 log.cc:826] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:42.926363 8942 log.cc:826] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:42.927917 8943 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: No bootstrap required, opened a new log
I20251026 21:06:42.928113 8943 ts_tablet_manager.cc:1403] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:06:42.928115 8942 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: No bootstrap required, opened a new log
I20251026 21:06:42.928115 8941 tablet_bootstrap.cc:492] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: No bootstrap required, opened a new log
I20251026 21:06:42.928274 8942 ts_tablet_manager.cc:1403] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:06:42.928274 8941 ts_tablet_manager.cc:1403] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 21:06:42.931259 8943 raft_consensus.cc:359] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.931526 8943 raft_consensus.cc:385] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:42.931587 8943 raft_consensus.cc:740] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589def50af714e74baa324d7683528c5, State: Initialized, Role: FOLLOWER
I20251026 21:06:42.931524 8942 raft_consensus.cc:359] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.931737 8942 raft_consensus.cc:385] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:42.931806 8942 raft_consensus.cc:740] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aabefc4fb92844308c2b26b48a573692, State: Initialized, Role: FOLLOWER
I20251026 21:06:42.931762 8941 raft_consensus.cc:359] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.931947 8941 raft_consensus.cc:385] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:42.932001 8941 raft_consensus.cc:740] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcca9a6ad5a240e4b221eca426911810, State: Initialized, Role: FOLLOWER
I20251026 21:06:42.932132 8943 consensus_queue.cc:260] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.932235 8942 consensus_queue.cc:260] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.932320 8941 consensus_queue.cc:260] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:42.932933 8654 heartbeater.cc:499] Master 127.3.175.62:34835 was elected leader, sending a full tablet report...
I20251026 21:06:42.933101 8941 ts_tablet_manager.cc:1434] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:06:42.933102 8943 ts_tablet_manager.cc:1434] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251026 21:06:42.933447 8789 heartbeater.cc:499] Master 127.3.175.62:34835 was elected leader, sending a full tablet report...
I20251026 21:06:42.933624 8924 heartbeater.cc:499] Master 127.3.175.62:34835 was elected leader, sending a full tablet report...
I20251026 21:06:42.934139 8942 ts_tablet_manager.cc:1434] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20251026 21:06:43.116401 8925 tablet.cc:2378] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:43.143723 8790 tablet.cc:2378] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:43.156634 8947 raft_consensus.cc:493] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:43.156853 8947 raft_consensus.cc:515] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
W20251026 21:06:43.157565 8655 tablet.cc:2378] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:43.158057 8947 leader_election.cc:290] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcca9a6ad5a240e4b221eca426911810 (127.3.175.3:40099), aabefc4fb92844308c2b26b48a573692 (127.3.175.1:44651)
I20251026 21:06:43.161212 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.158281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:40099 (local address 127.3.175.2:47201)
1026 21:06:43.158570 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:43.158587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:43.158773 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:43.159039 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:43.159042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:43.159062 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:43.159371 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:43.159378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.160116 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:43.160119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:43.160724 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:43.160731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.160846 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:43.160871 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:06:43.160926 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 21:06:43.160986 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"spinlock_wait_cycles":95104,"thread_start_us":85,"threads_started":1}
I20251026 21:06:43.161216 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.158294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44651 (local address 127.3.175.2:54487)
1026 21:06:43.158598 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:43.158609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:43.158767 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:43.159015 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:43.159018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:43.159035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:06:43.159333 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:43.159346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.160115 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:43.160119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:43.160725 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:43.160732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.160847 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:43.160871 (+ 24us) client_negotiation.cc:769] Sending connection context
1026 21:06:43.160925 (+ 54us) client_negotiation.cc:240] Negotiation successful
1026 21:06:43.160990 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"spinlock_wait_cycles":4224,"thread_start_us":79,"threads_started":1}
I20251026 21:06:43.161607 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.158382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:47201 (local address 127.3.175.3:40099)
1026 21:06:43.158528 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.158533 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.158693 (+ 160us) server_negotiation.cc:408] Connection header received
1026 21:06:43.158894 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.158896 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.158940 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.159018 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:43.159493 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.159978 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:43.160857 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.161359 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:43.161390 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.161437 (+ 47us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.161482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251026 21:06:43.161624 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.158391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:54487 (local address 127.3.175.1:44651)
1026 21:06:43.158592 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.158598 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.158679 (+ 81us) server_negotiation.cc:408] Connection header received
1026 21:06:43.158878 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.158881 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.158925 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.158985 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:06:43.159461 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.159980 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:43.160850 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.161347 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:43.161389 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.161439 (+ 50us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.161492 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20251026 21:06:43.162472 8878 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b378e421d3f47cb8056967af7694fc2" candidate_uuid: "589def50af714e74baa324d7683528c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcca9a6ad5a240e4b221eca426911810" is_pre_election: true
I20251026 21:06:43.162554 8608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b378e421d3f47cb8056967af7694fc2" candidate_uuid: "589def50af714e74baa324d7683528c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabefc4fb92844308c2b26b48a573692" is_pre_election: true
I20251026 21:06:43.162750 8878 raft_consensus.cc:2468] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589def50af714e74baa324d7683528c5 in term 0.
I20251026 21:06:43.162801 8608 raft_consensus.cc:2468] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589def50af714e74baa324d7683528c5 in term 0.
I20251026 21:06:43.163261 8679 leader_election.cc:304] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 589def50af714e74baa324d7683528c5, aabefc4fb92844308c2b26b48a573692, fcca9a6ad5a240e4b221eca426911810; no voters:
I20251026 21:06:43.163558 8947 raft_consensus.cc:2804] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:43.163636 8947 raft_consensus.cc:493] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:43.163671 8947 raft_consensus.cc:3060] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:43.164597 8947 raft_consensus.cc:515] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:43.164976 8947 leader_election.cc:290] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [CANDIDATE]: Term 1 election: Requested vote from peers fcca9a6ad5a240e4b221eca426911810 (127.3.175.3:40099), aabefc4fb92844308c2b26b48a573692 (127.3.175.1:44651)
I20251026 21:06:43.165334 8878 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b378e421d3f47cb8056967af7694fc2" candidate_uuid: "589def50af714e74baa324d7683528c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcca9a6ad5a240e4b221eca426911810"
I20251026 21:06:43.165376 8608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b378e421d3f47cb8056967af7694fc2" candidate_uuid: "589def50af714e74baa324d7683528c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabefc4fb92844308c2b26b48a573692"
I20251026 21:06:43.165441 8878 raft_consensus.cc:3060] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:43.165500 8608 raft_consensus.cc:3060] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:43.166447 8878 raft_consensus.cc:2468] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589def50af714e74baa324d7683528c5 in term 1.
I20251026 21:06:43.166478 8608 raft_consensus.cc:2468] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589def50af714e74baa324d7683528c5 in term 1.
I20251026 21:06:43.166721 8679 leader_election.cc:304] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589def50af714e74baa324d7683528c5, fcca9a6ad5a240e4b221eca426911810; no voters:
I20251026 21:06:43.166896 8947 raft_consensus.cc:2804] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:43.167119 8947 raft_consensus.cc:697] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 1 LEADER]: Becoming Leader. State: Replica: 589def50af714e74baa324d7683528c5, State: Running, Role: LEADER
I20251026 21:06:43.167479 8947 consensus_queue.cc:237] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } }
I20251026 21:06:43.170588 8460 catalog_manager.cc:5649] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 589def50af714e74baa324d7683528c5 (127.3.175.2). New cstate: current_term: 1 leader_uuid: "589def50af714e74baa324d7683528c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589def50af714e74baa324d7683528c5" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:43.206041 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.202849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.3.175.2:40465)
1026 21:06:43.203006 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.203014 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.203031 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:06:43.203133 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.203135 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.203192 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.203299 (+ 107us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:06:43.203687 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.204201 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:43.204867 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:43.205061 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:43.205171 (+ 110us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:06:43.205624 (+ 453us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:06:43.205742 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.205832 (+ 90us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.205891 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:06:43.211766 8608 raft_consensus.cc:1275] T 1b378e421d3f47cb8056967af7694fc2 P aabefc4fb92844308c2b26b48a573692 [term 1 FOLLOWER]: Refusing update from remote peer 589def50af714e74baa324d7683528c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:43.211766 8878 raft_consensus.cc:1275] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 1 FOLLOWER]: Refusing update from remote peer 589def50af714e74baa324d7683528c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:43.212462 8947 consensus_queue.cc:1048] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcca9a6ad5a240e4b221eca426911810" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:43.212612 8952 consensus_queue.cc:1048] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aabefc4fb92844308c2b26b48a573692" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:43.220836 8956 mvcc.cc:204] Tried to move back new op lower bound from 7215156441948184576 to 7215156441776799744. Current Snapshot: MvccSnapshot[applied={T|T < 7215156441948184576}]
I20251026 21:06:43.221119 8957 mvcc.cc:204] Tried to move back new op lower bound from 7215156441948184576 to 7215156441776799744. Current Snapshot: MvccSnapshot[applied={T|T < 7215156441948184576}]
I20251026 21:06:43.229290 8460 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35634:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:43.229460 8460 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35634:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:43.232034 8460 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3a3df75443c0494699d9a6ebcaddb181: Sending DeleteTablet for 3 replicas of tablet 1b378e421d3f47cb8056967af7694fc2
I20251026 21:06:43.233054 8723 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b378e421d3f47cb8056967af7694fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:43 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:53786
I20251026 21:06:43.233040 8588 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b378e421d3f47cb8056967af7694fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:43 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:39668
I20251026 21:06:43.233134 8858 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b378e421d3f47cb8056967af7694fc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:43 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:45762
I20251026 21:06:43.233837 8967 tablet_replica.cc:333] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: stopping tablet replica
I20251026 21:06:43.233834 8965 tablet_replica.cc:333] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: stopping tablet replica
I20251026 21:06:43.234120 8965 raft_consensus.cc:2243] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:06:43.234128 8967 raft_consensus.cc:2243] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:43.234472 8967 raft_consensus.cc:2272] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:43.234483 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8526
I20251026 21:06:43.234579 8965 raft_consensus.cc:2272] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:43.236060 8967 ts_tablet_manager.cc:1916] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:43.238240 8965 ts_tablet_manager.cc:1916] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:43.238746 8967 ts_tablet_manager.cc:1929] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:43.238845 8967 log.cc:1199] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/1b378e421d3f47cb8056967af7694fc2
I20251026 21:06:43.239174 8967 ts_tablet_manager.cc:1950] T 1b378e421d3f47cb8056967af7694fc2 P fcca9a6ad5a240e4b221eca426911810: Deleting consensus metadata
I20251026 21:06:43.240489 8449 catalog_manager.cc:4985] TS fcca9a6ad5a240e4b221eca426911810 (127.3.175.3:40099): tablet 1b378e421d3f47cb8056967af7694fc2 (table test-table [id=b1febb61a54d415694208aec83e8bead]) successfully deleted
I20251026 21:06:43.241793 8965 ts_tablet_manager.cc:1929] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:43.241894 8965 log.cc:1199] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/1b378e421d3f47cb8056967af7694fc2
I20251026 21:06:43.242226 8965 ts_tablet_manager.cc:1950] T 1b378e421d3f47cb8056967af7694fc2 P 589def50af714e74baa324d7683528c5: Deleting consensus metadata
I20251026 21:06:43.243500 8449 catalog_manager.cc:4985] TS 589def50af714e74baa324d7683528c5 (127.3.175.2:40465): tablet 1b378e421d3f47cb8056967af7694fc2 (table test-table [id=b1febb61a54d415694208aec83e8bead]) successfully deleted
W20251026 21:06:43.243897 8448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.175.1:44651 (error 108)
I20251026 21:06:43.244182 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8661
I20251026 21:06:43.245108 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.244798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44651 (local address 127.0.0.1:39678)
1026 21:06:43.244931 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:43.245006 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251026 21:06:43.245309 8448 catalog_manager.cc:4712] TS aabefc4fb92844308c2b26b48a573692 (127.3.175.1:44651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b378e421d3f47cb8056967af7694fc2: Network error: Client connection negotiation failed: client connection to 127.3.175.1:44651: connect: Connection refused (error 111)
I20251026 21:06:43.250680 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8796
I20251026 21:06:43.256336 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8431
2025-10-26T21:06:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3464 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-26T21:06:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:43Z Disabled control of system clock
I20251026 21:06:43.286365 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:38983
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 21:06:43.397915 8972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:43.398177 8972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:43.398227 8972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:43.401669 8972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:43.401744 8972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:43.401762 8972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:43.401780 8972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:43.401798 8972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:06:43.405820 8972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:38983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:43.407047 8972 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:43.408239 8972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:43.413805 8977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.413792 8978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.413792 8980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:43.414216 8972 server_base.cc:1047] running on GCE node
I20251026 21:06:43.414718 8972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:43.415702 8972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:43.416862 8972 hybrid_clock.cc:648] HybridClock initialized: now 1761512803416848 us; error 36 us; skew 500 ppm
I20251026 21:06:43.418685 8972 webserver.cc:492] Webserver started at http://127.3.175.62:44571/ using document root <none> and password file <none>
I20251026 21:06:43.419318 8972 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:43.419402 8972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:43.419606 8972 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:43.421367 8972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "4720d25a958a457fb2e315d5be91ff84"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "588d482b128d65900a79e54939566aed"
server_key_iv: "67eeb0412a4299c91cbe44a85b15e0c2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.421850 8972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "4720d25a958a457fb2e315d5be91ff84"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "588d482b128d65900a79e54939566aed"
server_key_iv: "67eeb0412a4299c91cbe44a85b15e0c2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.425328 8972 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:43.427541 8986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.428629 8972 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:43.428750 8972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "4720d25a958a457fb2e315d5be91ff84"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "588d482b128d65900a79e54939566aed"
server_key_iv: "67eeb0412a4299c91cbe44a85b15e0c2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.428854 8972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:43.433811 8972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:43.434404 8972 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:43.434564 8972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:43.441713 9038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:38983 every 8 connection(s)
I20251026 21:06:43.441713 8972 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:38983
I20251026 21:06:43.443027 8972 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:43.445917 9039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:43.451565 9039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: Bootstrap starting.
I20251026 21:06:43.452072 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 8972
I20251026 21:06:43.452198 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:43.452428 3772 external_mini_cluster.cc:1468] Setting key 72a7620138a74fba2053cf63137c40c7
I20251026 21:06:43.454234 9039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:43.455093 9039 log.cc:826] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:43.457348 9039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: No bootstrap required, opened a new log
I20251026 21:06:43.458163 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.453762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.3.175.62:38983)
1026 21:06:43.454596 (+ 834us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.454608 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.454649 (+ 41us) server_negotiation.cc:408] Connection header received
1026 21:06:43.455428 (+ 779us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.455459 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.455778 (+ 319us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.456114 (+ 336us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:43.456856 (+ 742us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:43.456862 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:43.456873 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:43.456885 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:43.456934 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:43.456950 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:43.457074 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.457189 (+ 115us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.457441 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":191,"threads_started":1}
I20251026 21:06:43.460445 9039 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } }
I20251026 21:06:43.460629 9039 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:43.460670 9039 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4720d25a958a457fb2e315d5be91ff84, State: Initialized, Role: FOLLOWER
I20251026 21:06:43.461024 9039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } }
I20251026 21:06:43.461131 9039 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:43.461177 9039 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:43.461242 9039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:43.462086 9039 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } }
I20251026 21:06:43.462397 9039 leader_election.cc:304] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4720d25a958a457fb2e315d5be91ff84; no voters:
I20251026 21:06:43.462644 9039 leader_election.cc:290] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:43.462855 9044 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:43.463085 9044 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [term 1 LEADER]: Becoming Leader. State: Replica: 4720d25a958a457fb2e315d5be91ff84, State: Running, Role: LEADER
I20251026 21:06:43.463431 9044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } }
I20251026 21:06:43.463816 9039 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:43.465020 9046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4720d25a958a457fb2e315d5be91ff84. Latest consensus state: current_term: 1 leader_uuid: "4720d25a958a457fb2e315d5be91ff84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } } }
I20251026 21:06:43.465144 9046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:43.465092 9045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4720d25a958a457fb2e315d5be91ff84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4720d25a958a457fb2e315d5be91ff84" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 38983 } } }
I20251026 21:06:43.465215 9045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:43.466620 9053 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:43.469251 9053 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:43.474510 9053 catalog_manager.cc:1357] Generated new cluster ID: 1423c8702d38414b803e67b1e3dba88f
I20251026 21:06:43.474596 9053 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:43.482945 9053 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:43.483333 9053 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:43.493021 9053 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: Generated new TSK 0
I20251026 21:06:43.493873 9053 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:06:43.500149 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--builtin_ntp_servers=127.3.175.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 21:06:43.611764 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:43.611997 9063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:43.612063 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:43.615671 9063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:43.615736 9063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:43.615816 9063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:06:43.619741 9063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:43.620846 9063 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:43.621991 9063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:43.629016 9068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:43.629148 9063 server_base.cc:1047] running on GCE node
W20251026 21:06:43.629073 9071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.629033 9069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:43.629662 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:43.630228 9063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:43.631374 9063 hybrid_clock.cc:648] HybridClock initialized: now 1761512803631354 us; error 34 us; skew 500 ppm
I20251026 21:06:43.633249 9063 webserver.cc:492] Webserver started at http://127.3.175.1:41331/ using document root <none> and password file <none>
I20251026 21:06:43.633854 9063 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:43.633911 9063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:43.634102 9063 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:43.635918 9063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "c58a69e777d842469390c6ded8ff93e0"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "072b5b96708eb288fea9911d8c5576e4"
server_key_iv: "42f65156502a66fb8d80ed194fa37d56"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.636515 9063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "c58a69e777d842469390c6ded8ff93e0"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "072b5b96708eb288fea9911d8c5576e4"
server_key_iv: "42f65156502a66fb8d80ed194fa37d56"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.641026 9063 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251026 21:06:43.643687 9077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.644727 9063 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:06:43.644837 9063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "c58a69e777d842469390c6ded8ff93e0"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "072b5b96708eb288fea9911d8c5576e4"
server_key_iv: "42f65156502a66fb8d80ed194fa37d56"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.644924 9063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:43.649968 9063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:43.650549 9063 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:43.650693 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:43.651211 9063 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:43.652186 9063 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:43.652233 9063 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.652273 9063 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:43.652295 9063 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.661608 9063 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:35777
I20251026 21:06:43.661638 9190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:35777 every 8 connection(s)
I20251026 21:06:43.662618 9063 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:06:43.666020 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9063
I20251026 21:06:43.666123 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:06:43.666352 3772 external_mini_cluster.cc:1468] Setting key 2d0171bc5aa498a2d483bb37a67f5cce
I20251026 21:06:43.668534 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.664413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:50489 (local address 127.3.175.62:38983)
1026 21:06:43.664554 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.664558 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.665549 (+ 991us) server_negotiation.cc:408] Connection header received
1026 21:06:43.666500 (+ 951us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.666504 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.666557 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.666659 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:43.668037 (+ 1378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:43.668042 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:43.668045 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:43.668057 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:43.668090 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:43.668101 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:43.668194 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.668329 (+ 135us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.668375 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:06:43.668565 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--builtin_ntp_servers=127.3.175.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251026 21:06:43.669293 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.664727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:38983 (local address 127.3.175.1:50489)
1026 21:06:43.665392 (+ 665us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:43.665428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:43.666281 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:43.666816 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:43.666842 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:43.666954 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:43.667702 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:43.667704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:43.667788 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:43.667864 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:43.668186 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:43.668192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:43.668195 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 21:06:43.668362 (+ 167us) client_negotiation.cc:240] Negotiation successful
1026 21:06:43.668544 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20251026 21:06:43.670854 9191 heartbeater.cc:344] Connected to a master server at 127.3.175.62:38983
I20251026 21:06:43.671135 9191 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:43.671595 9191 heartbeater.cc:507] Master 127.3.175.62:38983 requested a full tablet report, sending...
I20251026 21:06:43.673341 9003 ts_manager.cc:194] Registered new tserver with Master: c58a69e777d842469390c6ded8ff93e0 (127.3.175.1:35777)
W20251026 21:06:43.781651 9194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:43.781939 9194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:43.781993 9194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:43.785535 9194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:43.785616 9194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:43.785693 9194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:06:43.789610 9194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:43.790755 9194 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:43.791977 9194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:43.798839 9202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.798820 9199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.798820 9200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:43.799072 9194 server_base.cc:1047] running on GCE node
I20251026 21:06:43.799522 9194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:43.800139 9194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:43.801299 9194 hybrid_clock.cc:648] HybridClock initialized: now 1761512803801289 us; error 27 us; skew 500 ppm
I20251026 21:06:43.803182 9194 webserver.cc:492] Webserver started at http://127.3.175.2:40229/ using document root <none> and password file <none>
I20251026 21:06:43.803798 9194 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:43.803864 9194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:43.804078 9194 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:43.805771 9194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "2468e1e2ee7342f2b9bd258d146e2fb8"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "6e09a9ad0afea7da1e0990a4abe41155"
server_key_iv: "cd3afbbf3aae71853c142ff2c84bca50"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.806259 9194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "2468e1e2ee7342f2b9bd258d146e2fb8"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "6e09a9ad0afea7da1e0990a4abe41155"
server_key_iv: "cd3afbbf3aae71853c142ff2c84bca50"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.809811 9194 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:06:43.812215 9208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.813308 9194 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:43.813419 9194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "2468e1e2ee7342f2b9bd258d146e2fb8"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "6e09a9ad0afea7da1e0990a4abe41155"
server_key_iv: "cd3afbbf3aae71853c142ff2c84bca50"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.813515 9194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:43.818506 9194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:43.819190 9194 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:43.819382 9194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:43.819932 9194 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:43.820907 9194 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:43.820957 9194 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.821017 9194 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:43.821050 9194 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.830313 9194 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:40271
I20251026 21:06:43.830339 9321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:40271 every 8 connection(s)
I20251026 21:06:43.831362 9194 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:06:43.834708 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9194
I20251026 21:06:43.834846 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:06:43.835117 3772 external_mini_cluster.cc:1468] Setting key 4423838720d48df03423ba8e81ce3b7f
I20251026 21:06:43.837011 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.833055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:58281 (local address 127.3.175.62:38983)
1026 21:06:43.833215 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:43.833220 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:43.834281 (+ 1061us) server_negotiation.cc:408] Connection header received
1026 21:06:43.835320 (+ 1039us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:43.835325 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:43.835374 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:43.835451 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:43.836579 (+ 1128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:43.836583 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:43.836586 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:43.836598 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:43.836624 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:43.836632 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:43.836704 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:43.836871 (+ 167us) server_negotiation.cc:300] Negotiation successful
1026 21:06:43.836907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:06:43.837245 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--builtin_ntp_servers=127.3.175.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251026 21:06:43.838024 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:43.833469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:38983 (local address 127.3.175.2:58281)
1026 21:06:43.834146 (+ 677us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:43.834182 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:43.835071 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:43.835627 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:43.835636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:43.835725 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:43.836366 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:43.836368 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:43.836400 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:43.836454 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:43.836712 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:43.836720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:43.836725 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 21:06:43.836944 (+ 219us) client_negotiation.cc:240] Negotiation successful
1026 21:06:43.837194 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20251026 21:06:43.839368 9322 heartbeater.cc:344] Connected to a master server at 127.3.175.62:38983
I20251026 21:06:43.839624 9322 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:43.840109 9322 heartbeater.cc:507] Master 127.3.175.62:38983 requested a full tablet report, sending...
I20251026 21:06:43.841342 9003 ts_manager.cc:194] Registered new tserver with Master: 2468e1e2ee7342f2b9bd258d146e2fb8 (127.3.175.2:40271)
W20251026 21:06:43.950594 9325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:43.950867 9325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:43.950920 9325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:43.954365 9325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:43.954443 9325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:43.954524 9325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:06:43.958669 9325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:43.959920 9325 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:43.961077 9325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:43.968174 9330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.968161 9333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:43.968161 9331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:43.968568 9325 server_base.cc:1047] running on GCE node
I20251026 21:06:43.968979 9325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:43.969586 9325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:43.970835 9325 hybrid_clock.cc:648] HybridClock initialized: now 1761512803970812 us; error 44 us; skew 500 ppm
I20251026 21:06:43.972813 9325 webserver.cc:492] Webserver started at http://127.3.175.3:36119/ using document root <none> and password file <none>
I20251026 21:06:43.973409 9325 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:43.973479 9325 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:43.973671 9325 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:43.975564 9325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "35b1aed9524d48ecb0da88aaaa48fb3d"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "4c3736c89136e4b93b44aaee1b05e150"
server_key_iv: "85d4230eb72fe31ae9a55dac7c82f1d2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.976078 9325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "35b1aed9524d48ecb0da88aaaa48fb3d"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "4c3736c89136e4b93b44aaee1b05e150"
server_key_iv: "85d4230eb72fe31ae9a55dac7c82f1d2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.979856 9325 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20251026 21:06:43.982455 9339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.983618 9325 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:43.983742 9325 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "35b1aed9524d48ecb0da88aaaa48fb3d"
format_stamp: "Formatted at 2025-10-26 21:06:43 on dist-test-slave-th6x"
server_key: "4c3736c89136e4b93b44aaee1b05e150"
server_key_iv: "85d4230eb72fe31ae9a55dac7c82f1d2"
server_key_version: "encryptionkey@0"
I20251026 21:06:43.983893 9325 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:43.988941 9325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:43.989573 9325 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:43.989763 9325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:43.990294 9325 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:06:43.991256 9325 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:06:43.991359 9325 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:43.991425 9325 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:06:43.991452 9325 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:44.000892 9325 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:35661
I20251026 21:06:44.000914 9452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:35661 every 8 connection(s)
I20251026 21:06:44.001935 9325 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:06:44.003634 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9325
I20251026 21:06:44.003773 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:06:44.004096 3772 external_mini_cluster.cc:1468] Setting key 661d1ce2bb1cce93116e80c4312fcb7a
I20251026 21:06:44.011627 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.003823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:59963 (local address 127.3.175.62:38983)
1026 21:06:44.004005 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.004011 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.005068 (+ 1057us) server_negotiation.cc:408] Connection header received
1026 21:06:44.006039 (+ 971us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.006043 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.006092 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.006156 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.011149 (+ 4993us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.011153 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.011155 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.011170 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.011193 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.011201 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.011332 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.011474 (+ 142us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.011514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:06:44.012457 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.004119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:38983 (local address 127.3.175.3:59963)
1026 21:06:44.004915 (+ 796us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.004962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.005788 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.006322 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.006332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.006420 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.010844 (+ 4424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.010846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.010888 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.010959 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.011327 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.011335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.011339 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.011532 (+ 193us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.011756 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20251026 21:06:44.013856 9453 heartbeater.cc:344] Connected to a master server at 127.3.175.62:38983
I20251026 21:06:44.014096 9453 heartbeater.cc:461] Registering TS with master...
I20251026 21:06:44.014503 9453 heartbeater.cc:507] Master 127.3.175.62:38983 requested a full tablet report, sending...
I20251026 21:06:44.015597 9003 ts_manager.cc:194] Registered new tserver with Master: 35b1aed9524d48ecb0da88aaaa48fb3d (127.3.175.3:35661)
I20251026 21:06:44.017923 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:06:44.021414 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.019765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.3.175.62:38983)
1026 21:06:44.019909 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.019912 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.020033 (+ 121us) server_negotiation.cc:408] Connection header received
1026 21:06:44.020139 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.020142 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.020178 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.020246 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.021038 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.021040 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.021042 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.021050 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.021065 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.021071 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.021134 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.021276 (+ 142us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.021309 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251026 21:06:44.024144 9003 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:06:44.026660 9003 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:06:44.039073 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.035158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.3.175.1:35777)
1026 21:06:44.035562 (+ 404us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.035567 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.035956 (+ 389us) server_negotiation.cc:408] Connection header received
1026 21:06:44.036169 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.036177 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.036339 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.036436 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.038481 (+ 2045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.038497 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.038508 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.038524 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.038590 (+ 66us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.038608 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.038731 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.038789 (+ 58us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.038870 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20251026 21:06:44.039086 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.036279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.3.175.2:40271)
1026 21:06:44.036620 (+ 341us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.036626 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.036647 (+ 21us) server_negotiation.cc:408] Connection header received
1026 21:06:44.036704 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.036712 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.036864 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.037002 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.038359 (+ 1357us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.038365 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.038376 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.038388 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.038447 (+ 59us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.038461 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.038582 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.038878 (+ 296us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.038944 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20251026 21:06:44.039155 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.035740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:35661 (local address 127.0.0.1:44498)
1026 21:06:44.036040 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.036054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.036144 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.038007 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.038011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.038025 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.038514 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.038516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.038529 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.038551 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.038913 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.038916 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.038919 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.038974 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.039009 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20251026 21:06:44.039000 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.034984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:35777 (local address 127.0.0.1:60748)
1026 21:06:44.035791 (+ 807us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.035852 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.036022 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.037159 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.037162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.037219 (+ 57us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.037933 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.037937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.037978 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.038038 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.038701 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.038708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.038714 (+ 6us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.038773 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.038849 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":75,"threads_started":1}
I20251026 21:06:44.039431 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.036229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.3.175.3:35661)
1026 21:06:44.037652 (+ 1423us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.037659 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.037683 (+ 24us) server_negotiation.cc:408] Connection header received
1026 21:06:44.037745 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.037750 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.037873 (+ 123us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.037995 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.038696 (+ 701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.038701 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.038710 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.038723 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.038789 (+ 66us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.038804 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.039110 (+ 306us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.039154 (+ 44us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.039232 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":173,"threads_started":1}
I20251026 21:06:44.040359 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.035132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:40271 (local address 127.0.0.1:52866)
1026 21:06:44.035886 (+ 754us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.035901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.036005 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.037146 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.037153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.037217 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.038181 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.038185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.038200 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.038225 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.038620 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.038630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.038634 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.038754 (+ 120us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.040235 (+ 1481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":1394,"thread_start_us":113,"threads_started":1}
I20251026 21:06:44.042197 9125 tablet_service.cc:1505] Processing CreateTablet for tablet 926df1848c9c40daaed5b978bb467b6f (DEFAULT_TABLE table=test-table [id=3395a271ca0942e0a06c0b81828ea9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:44.042268 9387 tablet_service.cc:1505] Processing CreateTablet for tablet 926df1848c9c40daaed5b978bb467b6f (DEFAULT_TABLE table=test-table [id=3395a271ca0942e0a06c0b81828ea9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:44.043203 9387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926df1848c9c40daaed5b978bb467b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:44.043208 9125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926df1848c9c40daaed5b978bb467b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:44.042954 9256 tablet_service.cc:1505] Processing CreateTablet for tablet 926df1848c9c40daaed5b978bb467b6f (DEFAULT_TABLE table=test-table [id=3395a271ca0942e0a06c0b81828ea9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:06:44.043852 9256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926df1848c9c40daaed5b978bb467b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:44.048954 9470 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Bootstrap starting.
I20251026 21:06:44.050381 9472 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Bootstrap starting.
I20251026 21:06:44.050469 9471 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Bootstrap starting.
I20251026 21:06:44.051273 9470 tablet_bootstrap.cc:654] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:44.052162 9470 log.cc:826] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:44.052686 9472 tablet_bootstrap.cc:654] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:44.052686 9471 tablet_bootstrap.cc:654] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:44.053552 9472 log.cc:826] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:44.053552 9471 log.cc:826] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:44.054131 9470 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: No bootstrap required, opened a new log
I20251026 21:06:44.054356 9470 ts_tablet_manager.cc:1403] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 21:06:44.055569 9472 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: No bootstrap required, opened a new log
I20251026 21:06:44.055779 9472 ts_tablet_manager.cc:1403] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20251026 21:06:44.055939 9471 tablet_bootstrap.cc:492] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: No bootstrap required, opened a new log
I20251026 21:06:44.056160 9471 ts_tablet_manager.cc:1403] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20251026 21:06:44.057173 9470 raft_consensus.cc:359] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.057376 9470 raft_consensus.cc:385] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:44.057435 9470 raft_consensus.cc:740] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58a69e777d842469390c6ded8ff93e0, State: Initialized, Role: FOLLOWER
I20251026 21:06:44.057896 9470 consensus_queue.cc:260] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.058851 9470 ts_tablet_manager.cc:1434] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251026 21:06:44.059036 9472 raft_consensus.cc:359] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.059028 9471 raft_consensus.cc:359] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.059242 9191 heartbeater.cc:499] Master 127.3.175.62:38983 was elected leader, sending a full tablet report...
I20251026 21:06:44.059255 9472 raft_consensus.cc:385] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:44.059254 9471 raft_consensus.cc:385] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:44.059334 9472 raft_consensus.cc:740] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b1aed9524d48ecb0da88aaaa48fb3d, State: Initialized, Role: FOLLOWER
I20251026 21:06:44.059334 9471 raft_consensus.cc:740] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2468e1e2ee7342f2b9bd258d146e2fb8, State: Initialized, Role: FOLLOWER
I20251026 21:06:44.059778 9472 consensus_queue.cc:260] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.059778 9471 consensus_queue.cc:260] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.060467 9453 heartbeater.cc:499] Master 127.3.175.62:38983 was elected leader, sending a full tablet report...
I20251026 21:06:44.060668 9472 ts_tablet_manager.cc:1434] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:06:44.060671 9471 ts_tablet_manager.cc:1434] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20251026 21:06:44.060992 9322 heartbeater.cc:499] Master 127.3.175.62:38983 was elected leader, sending a full tablet report...
W20251026 21:06:44.082255 9323 tablet.cc:2378] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:06:44.163781 9192 tablet.cc:2378] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:44.246024 9476 raft_consensus.cc:493] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:06:44.246249 9476 raft_consensus.cc:515] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.247551 9476 leader_election.cc:290] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2468e1e2ee7342f2b9bd258d146e2fb8 (127.3.175.2:40271), 35b1aed9524d48ecb0da88aaaa48fb3d (127.3.175.3:35661)
I20251026 21:06:44.249917 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.247946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:34219 (local address 127.3.175.2:40271)
1026 21:06:44.248251 (+ 305us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.248259 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.248278 (+ 19us) server_negotiation.cc:408] Connection header received
1026 21:06:44.248422 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.248426 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.248471 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.248567 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.249460 (+ 893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.249467 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.249473 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.249492 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.249518 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.249531 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.249619 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.249699 (+ 80us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.249757 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20251026 21:06:44.250041 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.247946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:51919 (local address 127.3.175.3:35661)
1026 21:06:44.248097 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.248104 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.248287 (+ 183us) server_negotiation.cc:408] Connection header received
1026 21:06:44.248441 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.248444 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.248479 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.248545 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.249559 (+ 1014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.249562 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.249564 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.249575 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.249600 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.249611 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.249705 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.249848 (+ 143us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.249892 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:06:44.250201 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.247849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:40271 (local address 127.3.175.1:34219)
1026 21:06:44.248183 (+ 334us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.248195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.248307 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.248592 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.248596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.248612 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.249291 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.249293 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.249305 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.249327 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.249615 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.249618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.249621 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.249705 (+ 84us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.249766 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":4864,"thread_start_us":113,"threads_started":1}
I20251026 21:06:44.250434 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.247837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:35661 (local address 127.3.175.1:51919)
1026 21:06:44.248168 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.248187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.248321 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.248586 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.248589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.248603 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:06:44.249383 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:06:44.249388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:06:44.249405 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:06:44.249436 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:06:44.249703 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:06:44.249708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:06:44.249711 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 21:06:44.250185 (+ 474us) client_negotiation.cc:240] Negotiation successful
1026 21:06:44.250225 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"spinlock_wait_cycles":9216,"thread_start_us":90,"threads_started":1}
I20251026 21:06:44.250967 9276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926df1848c9c40daaed5b978bb467b6f" candidate_uuid: "c58a69e777d842469390c6ded8ff93e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" is_pre_election: true
I20251026 21:06:44.251060 9407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926df1848c9c40daaed5b978bb467b6f" candidate_uuid: "c58a69e777d842469390c6ded8ff93e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" is_pre_election: true
I20251026 21:06:44.251345 9276 raft_consensus.cc:2468] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58a69e777d842469390c6ded8ff93e0 in term 0.
I20251026 21:06:44.251350 9407 raft_consensus.cc:2468] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58a69e777d842469390c6ded8ff93e0 in term 0.
I20251026 21:06:44.251864 9079 leader_election.cc:304] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b1aed9524d48ecb0da88aaaa48fb3d, c58a69e777d842469390c6ded8ff93e0; no voters:
I20251026 21:06:44.252120 9476 raft_consensus.cc:2804] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:06:44.252213 9476 raft_consensus.cc:493] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:06:44.252274 9476 raft_consensus.cc:3060] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 0 FOLLOWER]: Advancing to term 1
W20251026 21:06:44.253082 9454 tablet.cc:2378] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:06:44.253463 9476 raft_consensus.cc:515] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.253856 9476 leader_election.cc:290] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [CANDIDATE]: Term 1 election: Requested vote from peers 2468e1e2ee7342f2b9bd258d146e2fb8 (127.3.175.2:40271), 35b1aed9524d48ecb0da88aaaa48fb3d (127.3.175.3:35661)
I20251026 21:06:44.254205 9276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926df1848c9c40daaed5b978bb467b6f" candidate_uuid: "c58a69e777d842469390c6ded8ff93e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8"
I20251026 21:06:44.254220 9407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926df1848c9c40daaed5b978bb467b6f" candidate_uuid: "c58a69e777d842469390c6ded8ff93e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d"
I20251026 21:06:44.254313 9407 raft_consensus.cc:3060] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:44.254333 9276 raft_consensus.cc:3060] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:44.255743 9276 raft_consensus.cc:2468] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58a69e777d842469390c6ded8ff93e0 in term 1.
I20251026 21:06:44.255750 9407 raft_consensus.cc:2468] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58a69e777d842469390c6ded8ff93e0 in term 1.
I20251026 21:06:44.256083 9078 leader_election.cc:304] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2468e1e2ee7342f2b9bd258d146e2fb8, c58a69e777d842469390c6ded8ff93e0; no voters:
I20251026 21:06:44.256259 9476 raft_consensus.cc:2804] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:44.256489 9476 raft_consensus.cc:697] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [term 1 LEADER]: Becoming Leader. State: Replica: c58a69e777d842469390c6ded8ff93e0, State: Running, Role: LEADER
I20251026 21:06:44.256798 9476 consensus_queue.cc:237] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } }
I20251026 21:06:44.259912 9003 catalog_manager.cc:5649] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c58a69e777d842469390c6ded8ff93e0 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "c58a69e777d842469390c6ded8ff93e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58a69e777d842469390c6ded8ff93e0" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 35777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 } health_report { overall_health: UNKNOWN } } }
I20251026 21:06:44.276043 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.274530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.3.175.1:35777)
1026 21:06:44.274672 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.274676 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.274688 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:06:44.274774 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.274777 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.274844 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.274915 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.275674 (+ 759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.275676 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.275678 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.275688 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.275707 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.275714 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.275785 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.275850 (+ 65us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.275884 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:06:44.282264 9276 raft_consensus.cc:1275] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 1 FOLLOWER]: Refusing update from remote peer c58a69e777d842469390c6ded8ff93e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:44.282264 9407 raft_consensus.cc:1275] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 1 FOLLOWER]: Refusing update from remote peer c58a69e777d842469390c6ded8ff93e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:06:44.282920 9481 consensus_queue.cc:1048] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2468e1e2ee7342f2b9bd258d146e2fb8" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 40271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:44.283095 9476 consensus_queue.cc:1048] T 926df1848c9c40daaed5b978bb467b6f P c58a69e777d842469390c6ded8ff93e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b1aed9524d48ecb0da88aaaa48fb3d" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 35661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:06:44.292205 9485 mvcc.cc:204] Tried to move back new op lower bound from 7215156446331645952 to 7215156446238605312. Current Snapshot: MvccSnapshot[applied={T|T < 7215156446331645952}]
I20251026 21:06:44.300450 9002 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33572:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:44.300604 9002 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33572:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:06:44.303125 9002 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4720d25a958a457fb2e315d5be91ff84: Sending DeleteTablet for 3 replicas of tablet 926df1848c9c40daaed5b978bb467b6f
I20251026 21:06:44.304023 9125 tablet_service.cc:1552] Processing DeleteTablet for tablet 926df1848c9c40daaed5b978bb467b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:44 UTC) from {username='slave'} at 127.0.0.1:60748
I20251026 21:06:44.304024 9256 tablet_service.cc:1552] Processing DeleteTablet for tablet 926df1848c9c40daaed5b978bb467b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:44 UTC) from {username='slave'} at 127.0.0.1:52866
I20251026 21:06:44.304032 9387 tablet_service.cc:1552] Processing DeleteTablet for tablet 926df1848c9c40daaed5b978bb467b6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:06:44 UTC) from {username='slave'} at 127.0.0.1:44498
I20251026 21:06:44.304585 9495 tablet_replica.cc:333] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: stopping tablet replica
I20251026 21:06:44.304869 9496 tablet_replica.cc:333] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: stopping tablet replica
I20251026 21:06:44.304872 9495 raft_consensus.cc:2243] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:44.305230 9495 raft_consensus.cc:2272] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:44.305230 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9063
I20251026 21:06:44.306216 9496 raft_consensus.cc:2243] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:06:44.306653 9496 raft_consensus.cc:2272] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:06:44.308104 9495 ts_tablet_manager.cc:1916] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:44.309350 9496 ts_tablet_manager.cc:1916] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:06:44.310738 9495 ts_tablet_manager.cc:1929] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:44.310846 9495 log.cc:1199] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/926df1848c9c40daaed5b978bb467b6f
I20251026 21:06:44.311184 9495 ts_tablet_manager.cc:1950] T 926df1848c9c40daaed5b978bb467b6f P 2468e1e2ee7342f2b9bd258d146e2fb8: Deleting consensus metadata
W20251026 21:06:44.311666 8988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.175.1:35777 (error 108)
I20251026 21:06:44.311996 9496 ts_tablet_manager.cc:1929] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:06:44.312069 9496 log.cc:1199] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/926df1848c9c40daaed5b978bb467b6f
I20251026 21:06:44.312359 9496 ts_tablet_manager.cc:1950] T 926df1848c9c40daaed5b978bb467b6f P 35b1aed9524d48ecb0da88aaaa48fb3d: Deleting consensus metadata
I20251026 21:06:44.312467 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9194
I20251026 21:06:44.312528 8987 catalog_manager.cc:4985] TS 2468e1e2ee7342f2b9bd258d146e2fb8 (127.3.175.2:40271): tablet 926df1848c9c40daaed5b978bb467b6f (table test-table [id=3395a271ca0942e0a06c0b81828ea9fa]) successfully deleted
I20251026 21:06:44.313448 8988 catalog_manager.cc:4985] TS 35b1aed9524d48ecb0da88aaaa48fb3d (127.3.175.3:35661): tablet 926df1848c9c40daaed5b978bb467b6f (table test-table [id=3395a271ca0942e0a06c0b81828ea9fa]) successfully deleted
I20251026 21:06:44.313670 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.313238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:35777 (local address 127.0.0.1:60758)
1026 21:06:44.313488 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.313571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20251026 21:06:44.313809 8988 catalog_manager.cc:4712] TS c58a69e777d842469390c6ded8ff93e0 (127.3.175.1:35777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 926df1848c9c40daaed5b978bb467b6f: Network error: Client connection negotiation failed: client connection to 127.3.175.1:35777: connect: Connection refused (error 111)
I20251026 21:06:44.318661 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9325
I20251026 21:06:44.324345 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 8972
2025-10-26T21:06:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1067 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251026 21:06:44.346143 3772 mini_oidc.cc:150] Starting JWKS server
I20251026 21:06:44.347584 3772 webserver.cc:492] Webserver started at https://127.0.0.1:42435/ using document root <none> and password file <none>
I20251026 21:06:44.347726 3772 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 21:06:44.347997 3772 webserver.cc:492] Webserver started at http://127.0.0.1:41329/ using document root <none> and password file <none>
2025-10-26T21:06:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:44Z Disabled control of system clock
I20251026 21:06:44.356204 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44901
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:44901
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42435/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/testchainca.cert with env {}
W20251026 21:06:44.464300 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:44.464552 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:44.464607 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:44.467900 9503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:44.467967 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:44.467983 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:44.468005 9503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:44.468043 9503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 21:06:44.468084 9503 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 21:06:44.468111 9503 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42435/jwks.json
I20251026 21:06:44.472517 9503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:44901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42435/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:44.473631 9503 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:44.474488 9503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:44.480168 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:44.480234 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:44.480170 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:44.480448 9503 server_base.cc:1047] running on GCE node
I20251026 21:06:44.480993 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:44.481904 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:44.483057 9503 hybrid_clock.cc:648] HybridClock initialized: now 1761512804483035 us; error 35 us; skew 500 ppm
I20251026 21:06:44.484845 9503 webserver.cc:492] Webserver started at http://127.3.175.62:44597/ using document root <none> and password file <none>
I20251026 21:06:44.485410 9503 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:44.485473 9503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:44.485669 9503 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:44.487426 9503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "07440c93a35f4fe1b714a8a044bebc15"
format_stamp: "Formatted at 2025-10-26 21:06:44 on dist-test-slave-th6x"
server_key: "cc19321b3757214dc57f6a63b313ae94"
server_key_iv: "f227b26a889b6371fc8d40f8393248f0"
server_key_version: "encryptionkey@0"
I20251026 21:06:44.487924 9503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "07440c93a35f4fe1b714a8a044bebc15"
format_stamp: "Formatted at 2025-10-26 21:06:44 on dist-test-slave-th6x"
server_key: "cc19321b3757214dc57f6a63b313ae94"
server_key_iv: "f227b26a889b6371fc8d40f8393248f0"
server_key_version: "encryptionkey@0"
I20251026 21:06:44.491183 9503 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:06:44.493479 9517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:44.494429 9503 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:06:44.494541 9503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "07440c93a35f4fe1b714a8a044bebc15"
format_stamp: "Formatted at 2025-10-26 21:06:44 on dist-test-slave-th6x"
server_key: "cc19321b3757214dc57f6a63b313ae94"
server_key_iv: "f227b26a889b6371fc8d40f8393248f0"
server_key_version: "encryptionkey@0"
I20251026 21:06:44.494627 9503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:44.505200 9503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:44.505919 9503 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:44.506126 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:44.513518 9569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:44901 every 8 connection(s)
I20251026 21:06:44.513520 9503 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:44901
I20251026 21:06:44.514614 9503 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:44.517472 9570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:44.521476 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9503
I20251026 21:06:44.521595 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:44.521826 3772 external_mini_cluster.cc:1468] Setting key e63318311d7d0b67ef554049993984be
I20251026 21:06:44.523275 9570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15: Bootstrap starting.
I20251026 21:06:44.526043 9570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:44.527002 9570 log.cc:826] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:44.529251 9570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15: No bootstrap required, opened a new log
I20251026 21:06:44.530596 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.523089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.3.175.62:44901)
1026 21:06:44.523831 (+ 742us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.523844 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.523885 (+ 41us) server_negotiation.cc:408] Connection header received
1026 21:06:44.524615 (+ 730us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.524643 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.525099 (+ 456us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.525466 (+ 367us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:44.525962 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.527168 (+ 1206us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:44.528021 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.528341 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:44.528794 (+ 453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:44.528816 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:44.528828 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:44.529308 (+ 480us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:44.529361 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:44.529379 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:44.529485 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.529638 (+ 153us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.529870 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":159,"threads_started":1}
I20251026 21:06:44.532318 9570 raft_consensus.cc:359] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } }
I20251026 21:06:44.532521 9570 raft_consensus.cc:385] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:44.532591 9570 raft_consensus.cc:740] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07440c93a35f4fe1b714a8a044bebc15, State: Initialized, Role: FOLLOWER
I20251026 21:06:44.533085 9570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } }
I20251026 21:06:44.533212 9570 raft_consensus.cc:399] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:44.533277 9570 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:44.533375 9570 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:44.533448 3772 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 21:06:44.534439 9570 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } }
I20251026 21:06:44.534793 9570 leader_election.cc:304] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07440c93a35f4fe1b714a8a044bebc15; no voters:
I20251026 21:06:44.535147 9570 leader_election.cc:290] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:44.535235 9575 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:44.535427 9575 raft_consensus.cc:697] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [term 1 LEADER]: Becoming Leader. State: Replica: 07440c93a35f4fe1b714a8a044bebc15, State: Running, Role: LEADER
I20251026 21:06:44.535706 9575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } }
I20251026 21:06:44.536358 9570 sys_catalog.cc:565] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:44.537371 9577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07440c93a35f4fe1b714a8a044bebc15. Latest consensus state: current_term: 1 leader_uuid: "07440c93a35f4fe1b714a8a044bebc15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } } }
I20251026 21:06:44.537505 9577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:44.537865 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07440c93a35f4fe1b714a8a044bebc15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07440c93a35f4fe1b714a8a044bebc15" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 44901 } } }
I20251026 21:06:44.537953 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:44.537912 9584 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:44.540766 9584 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:44.546460 9584 catalog_manager.cc:1357] Generated new cluster ID: 8caef14468f24256bb8452ed673623cb
I20251026 21:06:44.546542 9584 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:44.556242 9584 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:44.557555 9584 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:44.568094 9584 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 07440c93a35f4fe1b714a8a044bebc15: Generated new TSK 0
I20251026 21:06:44.569119 9584 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:06:44.615700 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.583365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.3.175.62:44901)
1026 21:06:44.583512 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.583517 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.583610 (+ 93us) server_negotiation.cc:408] Connection header received
1026 21:06:44.583853 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.583857 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.583923 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.584051 (+ 128us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:06:44.584443 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.585292 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:44.586003 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.586206 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:44.586335 (+ 129us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:06:44.615150 (+ 28815us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1026 21:06:44.615269 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:44.615472 (+ 203us) server_negotiation.cc:300] Negotiation successful
1026 21:06:44.615533 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"thread_start_us":145,"threads_started":1}
I20251026 21:06:44.627475 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:44.623819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.3.175.62:44901)
1026 21:06:44.623962 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:44.623968 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:44.624067 (+ 99us) server_negotiation.cc:408] Connection header received
1026 21:06:44.624199 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:44.624201 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:44.624247 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:44.624312 (+ 65us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:06:44.624747 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.625263 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:44.626130 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.626263 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:44.626305 (+ 42us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:06:44.627134 (+ 829us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1026 21:06:44.627372 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60874: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":55}
W20251026 21:06:44.627553 9573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60874: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251026 21:06:44.628366 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:44.623691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44901 (local address 127.0.0.1:60874)
1026 21:06:44.623973 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:44.623996 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:44.624112 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:44.624335 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:44.624339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:44.624346 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1026 21:06:44.624637 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:44.624646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.625376 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:44.625381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:44.626030 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:44.626041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:44.626154 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:44.626179 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 21:06:44.627336 (+ 1157us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1026 21:06:44.627505 (+ 169us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.175.62:44901: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":136,"threads_started":1}
I20251026 21:06:59.638676 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:59.634405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.3.175.62:44901)
1026 21:06:59.634642 (+ 237us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:59.634645 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:59.634661 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:06:59.634884 (+ 223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:59.634887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:59.634937 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:59.635035 (+ 98us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:06:59.635521 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.636321 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:59.637208 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.637423 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:59.637470 (+ 47us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:06:59.638330 (+ 860us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1026 21:06:59.638517 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60224: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
W20251026 21:06:59.638753 9612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60224: JWT verification failed: token expired
W20251026 21:06:59.638811 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:59.634222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:44901 (local address 127.0.0.1:60224)
1026 21:06:59.634579 (+ 357us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:59.634596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:59.634729 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:59.635054 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:59.635057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:59.635064 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1026 21:06:59.635387 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:59.635399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.636462 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:59.636466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:59.637077 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:59.637088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.637217 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:59.637245 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 21:06:59.638461 (+ 1216us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1026 21:06:59.638635 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.175.62:44901: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":125,"threads_started":1}
W20251026 21:06:59.641727 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:59.640761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.3.175.62:44901)
1026 21:06:59.640889 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:59.640896 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:59.641238 (+ 342us) server_negotiation.cc:408] Connection header received
1026 21:06:59.641374 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:59.641377 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:59.641425 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:59.641491 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:59.641620 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60234: BlockingRecv error: recv got EOF from 127.0.0.1:60234 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20251026 21:06:59.645442 9622 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.3.175.62:44901
W20251026 21:06:59.646050 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:59.645159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.3.175.62:44901)
1026 21:06:59.645295 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:59.645299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:59.645395 (+ 96us) server_negotiation.cc:408] Connection header received
1026 21:06:59.645614 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:59.645617 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:59.645660 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:59.645730 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:59.645939 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60238: BlockingRecv error: recv got EOF from 127.0.0.1:60238 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 21:06:59.646206 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9503
2025-10-26T21:06:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251026 21:06:59.686928 3772 mini_oidc.cc:150] Starting JWKS server
I20251026 21:06:59.687904 3772 webserver.cc:492] Webserver started at https://127.0.0.1:44693/ using document root <none> and password file <none>
I20251026 21:06:59.687988 3772 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 21:06:59.688194 3772 webserver.cc:492] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
2025-10-26T21:06:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:59Z Disabled control of system clock
I20251026 21:06:59.696429 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46251
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:46251
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44693/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/testchainca.cert with env {}
W20251026 21:06:59.804061 9628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:06:59.804332 9628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:06:59.804381 9628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:06:59.807651 9628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:06:59.807721 9628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:06:59.807737 9628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:06:59.807757 9628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:06:59.807775 9628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 21:06:59.807791 9628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 21:06:59.807806 9628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44693/jwks.json
I20251026 21:06:59.812299 9628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44693/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:06:59.813390 9628 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:06:59.814229 9628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:06:59.820329 9634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:59.820367 9636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:06:59.820482 9633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:06:59.820593 9628 server_base.cc:1047] running on GCE node
I20251026 21:06:59.821136 9628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:06:59.822090 9628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:06:59.823244 9628 hybrid_clock.cc:648] HybridClock initialized: now 1761512819823235 us; error 25 us; skew 500 ppm
I20251026 21:06:59.825284 9628 webserver.cc:492] Webserver started at http://127.3.175.62:46695/ using document root <none> and password file <none>
I20251026 21:06:59.825865 9628 fs_manager.cc:362] Metadata directory not provided
I20251026 21:06:59.825924 9628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:06:59.826117 9628 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:06:59.827863 9628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "20cab1d90cd64e4dbd8751edc837eb74"
format_stamp: "Formatted at 2025-10-26 21:06:59 on dist-test-slave-th6x"
server_key: "072ddf2a6b58cea5127aa803b757367a"
server_key_iv: "45dd063d7d8927c28856f75c73d80cf4"
server_key_version: "encryptionkey@0"
I20251026 21:06:59.828347 9628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "20cab1d90cd64e4dbd8751edc837eb74"
format_stamp: "Formatted at 2025-10-26 21:06:59 on dist-test-slave-th6x"
server_key: "072ddf2a6b58cea5127aa803b757367a"
server_key_iv: "45dd063d7d8927c28856f75c73d80cf4"
server_key_version: "encryptionkey@0"
I20251026 21:06:59.831835 9628 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:06:59.834232 9642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:06:59.835276 9628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:06:59.835424 9628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "20cab1d90cd64e4dbd8751edc837eb74"
format_stamp: "Formatted at 2025-10-26 21:06:59 on dist-test-slave-th6x"
server_key: "072ddf2a6b58cea5127aa803b757367a"
server_key_iv: "45dd063d7d8927c28856f75c73d80cf4"
server_key_version: "encryptionkey@0"
I20251026 21:06:59.835513 9628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:06:59.855810 9628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:06:59.856442 9628 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:06:59.856588 9628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:06:59.864205 9628 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:46251
I20251026 21:06:59.864252 9694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:46251 every 8 connection(s)
I20251026 21:06:59.865306 9628 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:06:59.868342 9695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:06:59.871976 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9628
I20251026 21:06:59.872105 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:06:59.872376 3772 external_mini_cluster.cc:1468] Setting key 2d07f5004172e48f385082299d7d1c50
I20251026 21:06:59.874789 9695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74: Bootstrap starting.
I20251026 21:06:59.877230 9695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74: Neither blocks nor log segments found. Creating new log.
I20251026 21:06:59.877997 9695 log.cc:826] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74: Log is configured to *not* fsync() on all Append() calls
I20251026 21:06:59.880213 9695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74: No bootstrap required, opened a new log
I20251026 21:06:59.880509 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:59.873779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.3.175.62:46251)
1026 21:06:59.874467 (+ 688us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:59.874476 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:59.874511 (+ 35us) server_negotiation.cc:408] Connection header received
1026 21:06:59.875062 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:59.875082 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:59.875498 (+ 416us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:59.875810 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:06:59.876309 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.877075 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:59.877798 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.878087 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:59.878636 (+ 549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:06:59.878659 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:06:59.878670 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:06:59.879131 (+ 461us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:06:59.879183 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:06:59.879203 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:06:59.879333 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:06:59.879528 (+ 195us) server_negotiation.cc:300] Negotiation successful
1026 21:06:59.879769 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":170,"threads_started":1}
I20251026 21:06:59.883267 9695 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } }
I20251026 21:06:59.883536 9695 raft_consensus.cc:385] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:06:59.883587 9695 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cab1d90cd64e4dbd8751edc837eb74, State: Initialized, Role: FOLLOWER
I20251026 21:06:59.883594 3772 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 21:06:59.884048 9695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } }
I20251026 21:06:59.884178 9695 raft_consensus.cc:399] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:06:59.884239 9695 raft_consensus.cc:493] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:06:59.884328 9695 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:06:59.885340 9695 raft_consensus.cc:515] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } }
I20251026 21:06:59.885677 9695 leader_election.cc:304] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20cab1d90cd64e4dbd8751edc837eb74; no voters:
I20251026 21:06:59.885967 9695 leader_election.cc:290] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:06:59.886169 9700 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:06:59.886372 9700 raft_consensus.cc:697] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 20cab1d90cd64e4dbd8751edc837eb74, State: Running, Role: LEADER
I20251026 21:06:59.886679 9700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } }
I20251026 21:06:59.887169 9695 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:06:59.888487 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cab1d90cd64e4dbd8751edc837eb74. Latest consensus state: current_term: 1 leader_uuid: "20cab1d90cd64e4dbd8751edc837eb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } } }
I20251026 21:06:59.888514 9701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20cab1d90cd64e4dbd8751edc837eb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cab1d90cd64e4dbd8751edc837eb74" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46251 } } }
I20251026 21:06:59.888638 9701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:59.888638 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74 [sys.catalog]: This master's current role is: LEADER
I20251026 21:06:59.889191 9709 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:06:59.892197 9709 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:06:59.897728 9709 catalog_manager.cc:1357] Generated new cluster ID: dcbfd25dbfae4d37aaca458695e52f88
I20251026 21:06:59.897826 9709 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:06:59.907999 9709 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:06:59.909255 9709 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:06:59.917292 9709 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 20cab1d90cd64e4dbd8751edc837eb74: Generated new TSK 0
I20251026 21:06:59.918220 9709 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251026 21:06:59.945264 9725 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251026 21:06:59.945349 9725 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251026 21:06:59.946146 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:06:59.929344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.3.175.62:46251)
1026 21:06:59.929483 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 21:06:59.929488 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:06:59.929576 (+ 88us) server_negotiation.cc:408] Connection header received
1026 21:06:59.929736 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:06:59.929740 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:06:59.929813 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:06:59.929934 (+ 121us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:06:59.930359 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.931178 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:06:59.931990 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.932199 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:59.932322 (+ 123us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:06:59.945678 (+ 13356us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 21:06:59.945989 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20251026 21:06:59.946241 9698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251026 21:06:59.946249 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:06:59.929223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46251 (local address 127.0.0.1:56170)
1026 21:06:59.929487 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1026 21:06:59.929500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:06:59.929596 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:06:59.929972 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:06:59.929975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:06:59.929983 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1026 21:06:59.930235 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:59.930243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.931382 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:06:59.931385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:06:59.931862 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:06:59.931879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:06:59.931990 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:06:59.932012 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 21:06:59.945932 (+ 13920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 21:06:59.946093 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.175.62:46251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20251026 21:06:59.947081 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9628
2025-10-26T21:06:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251026 21:06:59.986807 3772 mini_oidc.cc:150] Starting JWKS server
I20251026 21:06:59.987787 3772 webserver.cc:492] Webserver started at https://127.0.0.1:44097/ using document root <none> and password file <none>
I20251026 21:06:59.987890 3772 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 21:06:59.988106 3772 webserver.cc:492] Webserver started at http://127.0.0.1:40763/ using document root <none> and password file <none>
2025-10-26T21:06:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:06:59Z Disabled control of system clock
I20251026 21:06:59.996125 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:35499
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:35499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251026 21:07:00.105783 9731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:00.106066 9731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:00.106122 9731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:00.109474 9731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:00.109552 9731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:00.109568 9731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:00.109594 9731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:00.109611 9731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 21:07:00.109629 9731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 21:07:00.109643 9731 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44097/jwks.json
I20251026 21:07:00.113906 9731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:35499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:00.115087 9731 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:00.116266 9731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:00.122043 9737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:00.122032 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:00.122032 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:00.122157 9731 server_base.cc:1047] running on GCE node
I20251026 21:07:00.122843 9731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:00.123873 9731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:00.125026 9731 hybrid_clock.cc:648] HybridClock initialized: now 1761512820125017 us; error 45 us; skew 500 ppm
I20251026 21:07:00.126881 9731 webserver.cc:492] Webserver started at http://127.3.175.62:40795/ using document root <none> and password file <none>
I20251026 21:07:00.127518 9731 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:00.127586 9731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:00.127772 9731 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:00.129541 9731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "63e5dc59f3f64ef393fbf6a1ce115343"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "da54416e7cc8db0768a8b3fcd6e934f8"
server_key_iv: "6c19f140fec07404c9546c9547b08ea1"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.130038 9731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "63e5dc59f3f64ef393fbf6a1ce115343"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "da54416e7cc8db0768a8b3fcd6e934f8"
server_key_iv: "6c19f140fec07404c9546c9547b08ea1"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.133570 9731 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251026 21:07:00.135871 9745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:00.136936 9731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:00.137058 9731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "63e5dc59f3f64ef393fbf6a1ce115343"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "da54416e7cc8db0768a8b3fcd6e934f8"
server_key_iv: "6c19f140fec07404c9546c9547b08ea1"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.137166 9731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:00.153378 9731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:00.154130 9731 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:00.154314 9731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:00.161773 9797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:35499 every 8 connection(s)
I20251026 21:07:00.161775 9731 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:35499
I20251026 21:07:00.162896 9731 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:00.166007 9798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:00.171746 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9731
I20251026 21:07:00.171890 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:00.171913 9798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343: Bootstrap starting.
I20251026 21:07:00.172111 3772 external_mini_cluster.cc:1468] Setting key f07e6b4456e2f12d428299d6fcc31ed2
I20251026 21:07:00.174501 9798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:00.175254 9798 log.cc:826] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:00.177150 9798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343: No bootstrap required, opened a new log
I20251026 21:07:00.180199 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:00.173338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.3.175.62:35499)
1026 21:07:00.174038 (+ 700us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:00.174047 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:00.174079 (+ 32us) server_negotiation.cc:408] Connection header received
1026 21:07:00.174652 (+ 573us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:00.174673 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:00.175032 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:00.175392 (+ 360us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:00.175878 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.176595 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.177262 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.177548 (+ 286us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.178144 (+ 596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:00.178173 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:00.178186 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:00.178658 (+ 472us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:00.178715 (+ 57us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:00.178734 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:00.178843 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:00.179117 (+ 274us) server_negotiation.cc:300] Negotiation successful
1026 21:07:00.179428 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20251026 21:07:00.180783 9798 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } }
I20251026 21:07:00.181011 9798 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:00.181068 9798 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e5dc59f3f64ef393fbf6a1ce115343, State: Initialized, Role: FOLLOWER
I20251026 21:07:00.181532 9798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } }
I20251026 21:07:00.181674 9798 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:00.181742 9798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:00.181852 9798 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:00.182894 9798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } }
I20251026 21:07:00.183322 3772 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 21:07:00.183306 9798 leader_election.cc:304] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e5dc59f3f64ef393fbf6a1ce115343; no voters:
I20251026 21:07:00.183650 9798 leader_election.cc:290] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:00.183758 9803 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:00.183979 9803 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [term 1 LEADER]: Becoming Leader. State: Replica: 63e5dc59f3f64ef393fbf6a1ce115343, State: Running, Role: LEADER
I20251026 21:07:00.184278 9803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } }
I20251026 21:07:00.184885 9798 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:00.185878 9804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } } }
I20251026 21:07:00.186053 9804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:00.185895 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e5dc59f3f64ef393fbf6a1ce115343. Latest consensus state: current_term: 1 leader_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e5dc59f3f64ef393fbf6a1ce115343" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 35499 } } }
I20251026 21:07:00.186367 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:00.186446 9812 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:00.189136 9812 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:00.194176 9812 catalog_manager.cc:1357] Generated new cluster ID: 1a9938dc5912470484cc738cff5627dd
I20251026 21:07:00.194264 9812 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:00.201779 9812 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:00.202693 9812 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:00.215574 9812 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 63e5dc59f3f64ef393fbf6a1ce115343: Generated new TSK 0
I20251026 21:07:00.216553 9812 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251026 21:07:00.250999 9828 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251026 21:07:00.251088 9828 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251026 21:07:00.252504 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:00.230650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.3.175.62:35499)
1026 21:07:00.230785 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:00.230789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:00.230898 (+ 109us) server_negotiation.cc:408] Connection header received
1026 21:07:00.231068 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:00.231071 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:00.231149 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:00.231237 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:07:00.231734 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.232596 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.233571 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.233714 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.233806 (+ 92us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:07:00.252074 (+ 18268us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 21:07:00.252389 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44056: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20251026 21:07:00.252566 9801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44056: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251026 21:07:00.252743 9827 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:00.230487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:35499 (local address 127.0.0.1:44056)
1026 21:07:00.230808 (+ 321us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:00.230824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:00.230942 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:00.231276 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:00.231305 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:00.231314 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1026 21:07:00.231610 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:00.231620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.232743 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.232748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:00.233454 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:00.233465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.233605 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.233633 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 21:07:00.252328 (+ 18695us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 21:07:00.252522 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.175.62:35499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
I20251026 21:07:00.253356 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9731
2025-10-26T21:07:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-26T21:07:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:00Z Disabled control of system clock
I20251026 21:07:00.301461 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34049
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:34049
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/testchainca.cert with env {}
W20251026 21:07:00.410583 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:00.410848 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:00.410938 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:00.414393 9832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:00.414476 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:00.414503 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:00.414537 9832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:00.414577 9832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 21:07:00.414620 9832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 21:07:00.414654 9832 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251026 21:07:00.419317 9832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:34049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:00.420498 9832 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:00.421401 9832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:00.427345 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:00.427269 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:00.427274 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:00.427572 9832 server_base.cc:1047] running on GCE node
I20251026 21:07:00.428119 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:00.429049 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:00.430207 9832 hybrid_clock.cc:648] HybridClock initialized: now 1761512820430181 us; error 41 us; skew 500 ppm
I20251026 21:07:00.432094 9832 webserver.cc:492] Webserver started at http://127.3.175.62:41679/ using document root <none> and password file <none>
I20251026 21:07:00.432706 9832 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:00.432765 9832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:00.432960 9832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:00.434728 9832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "d1a67a793062468b83c256f577efd522"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "07b1d00b248f2cdeac78f12f1a1b9ff5"
server_key_iv: "9471132a8a922d9fb1abd3097556cf80"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.435209 9832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "d1a67a793062468b83c256f577efd522"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "07b1d00b248f2cdeac78f12f1a1b9ff5"
server_key_iv: "9471132a8a922d9fb1abd3097556cf80"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.438537 9832 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:00.440855 9846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:00.441887 9832 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:00.442005 9832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "d1a67a793062468b83c256f577efd522"
format_stamp: "Formatted at 2025-10-26 21:07:00 on dist-test-slave-th6x"
server_key: "07b1d00b248f2cdeac78f12f1a1b9ff5"
server_key_iv: "9471132a8a922d9fb1abd3097556cf80"
server_key_version: "encryptionkey@0"
I20251026 21:07:00.442103 9832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:00.496001 9832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:00.496731 9832 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:00.496928 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:00.504503 9898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:34049 every 8 connection(s)
I20251026 21:07:00.504505 9832 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:34049
I20251026 21:07:00.505637 9832 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:00.507889 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9832
I20251026 21:07:00.508019 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:00.508301 3772 external_mini_cluster.cc:1468] Setting key 2d9bfa210ea506f48652db053031b5df
I20251026 21:07:00.508888 9899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:00.515570 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522: Bootstrap starting.
I20251026 21:07:00.516541 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:00.509513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.3.175.62:34049)
1026 21:07:00.510186 (+ 673us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:00.510195 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:00.510229 (+ 34us) server_negotiation.cc:408] Connection header received
1026 21:07:00.510878 (+ 649us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:00.510897 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:00.511249 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:00.511581 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:00.512281 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.513191 (+ 910us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.514073 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.514270 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.514895 (+ 625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:00.514941 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:00.514951 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:00.515393 (+ 442us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:00.515436 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:00.515452 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:00.515535 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:00.515707 (+ 172us) server_negotiation.cc:300] Negotiation successful
1026 21:07:00.515933 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":110,"threads_started":1}
I20251026 21:07:00.517930 9899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:00.518733 9899 log.cc:826] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:00.519506 3772 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 21:07:00.520790 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522: No bootstrap required, opened a new log
I20251026 21:07:00.523416 9899 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } }
I20251026 21:07:00.523681 9899 raft_consensus.cc:385] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:00.523741 9899 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a67a793062468b83c256f577efd522, State: Initialized, Role: FOLLOWER
I20251026 21:07:00.524178 9899 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } }
I20251026 21:07:00.524297 9899 raft_consensus.cc:399] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:00.524338 9899 raft_consensus.cc:493] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:00.524405 9899 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:00.525236 9899 raft_consensus.cc:515] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } }
I20251026 21:07:00.525547 9899 leader_election.cc:304] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a67a793062468b83c256f577efd522; no voters:
I20251026 21:07:00.525806 9899 leader_election.cc:290] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:00.525933 9904 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:00.526140 9904 raft_consensus.cc:697] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [term 1 LEADER]: Becoming Leader. State: Replica: d1a67a793062468b83c256f577efd522, State: Running, Role: LEADER
I20251026 21:07:00.526456 9904 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } }
I20251026 21:07:00.526945 9899 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:00.528085 9906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a67a793062468b83c256f577efd522. Latest consensus state: current_term: 1 leader_uuid: "d1a67a793062468b83c256f577efd522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } } }
I20251026 21:07:00.528213 9906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:00.528615 9913 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:00.529134 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a67a793062468b83c256f577efd522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a67a793062468b83c256f577efd522" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34049 } } }
I20251026 21:07:00.529232 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:00.531828 9913 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:00.537649 9913 catalog_manager.cc:1357] Generated new cluster ID: 252f890749864a51807859c4691e9ebe
I20251026 21:07:00.537730 9913 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:00.543658 9913 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:00.544593 9913 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:00.554742 9913 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d1a67a793062468b83c256f577efd522: Generated new TSK 0
I20251026 21:07:00.555606 9913 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:07:00.623138 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:00.566299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.3.175.62:34049)
1026 21:07:00.566437 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:00.566442 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:00.566549 (+ 107us) server_negotiation.cc:408] Connection header received
1026 21:07:00.566727 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:00.566729 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:00.566799 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:00.566923 (+ 124us) server_negotiation.cc:227] Negotiated authn=JWT
1026 21:07:00.567382 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.567910 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.568818 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.568966 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.569071 (+ 105us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 21:07:00.622618 (+ 53547us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1026 21:07:00.622983 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20251026 21:07:00.623235 9902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251026 21:07:00.623346 9928 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:00.566158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34049 (local address 127.0.0.1:56614)
1026 21:07:00.566445 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:00.566462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:00.566586 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:00.566939 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:00.566942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:00.566951 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1026 21:07:00.567244 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:00.567253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.568028 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:00.568032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:00.568701 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:00.568712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:00.568835 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:00.568861 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 21:07:00.622909 (+ 54048us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1026 21:07:00.623115 (+ 206us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.175.62:34049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20251026 21:07:00.624091 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9832
2025-10-26T21:07:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (350 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:00 dist-test-slave-th6x krb5kdc[9940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:00 dist-test-slave-th6x krb5kdc[9940](info): set up 2 sockets
Oct 26 21:07:00 dist-test-slave-th6x krb5kdc[9940](info): commencing operation
krb5kdc: starting...
W20251026 21:07:02.864148 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:02 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:03.005870 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:37023
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:03.116312 9956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:03.116567 9956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:03.116623 9956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:03.119959 9956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:03.120043 9956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:03.120066 9956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:03.120085 9956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:03.120102 9956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:03.124531 9956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:37023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:03.125667 9956 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:03.126513 9956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:03.132452 9964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.132447 9962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.132536 9961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:03.132905 9956 server_base.cc:1047] running on GCE node
I20251026 21:07:03.133426 9956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:03.134375 9956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:03.135524 9956 hybrid_clock.cc:648] HybridClock initialized: now 1761512823135512 us; error 29 us; skew 500 ppm
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:03.138203 9956 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:07:03.139366 9956 webserver.cc:492] Webserver started at http://127.3.175.62:44571/ using document root <none> and password file <none>
I20251026 21:07:03.139915 9956 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:03.139964 9956 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:03.140138 9956 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:03.141882 9956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "86438bcb34224ed783aa705830acc7bf"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "22c3f917f244d6bd9b3429b833a4366c"
server_key_iv: "eb9660a0e077d39d68c8b30c140ef738"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.142331 9956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "86438bcb34224ed783aa705830acc7bf"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "22c3f917f244d6bd9b3429b833a4366c"
server_key_iv: "eb9660a0e077d39d68c8b30c140ef738"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.146004 9956 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:07:03.148392 9971 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.149461 9956 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:03.149582 9956 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "86438bcb34224ed783aa705830acc7bf"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "22c3f917f244d6bd9b3429b833a4366c"
server_key_iv: "eb9660a0e077d39d68c8b30c140ef738"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.149688 9956 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:03.193396 9956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:03.199753 9956 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:03.199940 9956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:03.207269 10023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:37023 every 8 connection(s)
I20251026 21:07:03.207266 9956 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:37023
I20251026 21:07:03.208444 9956 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:03.211386 10024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:03.213070 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 9956
I20251026 21:07:03.213204 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:03.213464 3772 external_mini_cluster.cc:1468] Setting key 08e9d33dd86efc97b11e0392198e1c46
I20251026 21:07:03.217746 10024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf: Bootstrap starting.
I20251026 21:07:03.220037 10024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:03.220753 10024 log.cc:826] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf: Log is configured to *not* fsync() on all Append() calls
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:03.222857 10024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf: No bootstrap required, opened a new log
I20251026 21:07:03.225976 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.214879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.3.175.62:37023)
1026 21:07:03.215381 (+ 502us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:03.215391 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:03.215428 (+ 37us) server_negotiation.cc:408] Connection header received
1026 21:07:03.215937 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:03.215954 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:03.216269 (+ 315us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:03.216581 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:03.217521 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.218334 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.218987 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.219348 (+ 361us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.221587 (+ 2239us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:03.221605 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:03.221616 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:03.221664 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:03.223539 (+ 1875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.224007 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.224011 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.224015 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.224087 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.224336 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.224339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.224340 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.224664 (+ 324us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:03.224807 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:03.225089 (+ 282us) server_negotiation.cc:300] Negotiation successful
1026 21:07:03.225345 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
I20251026 21:07:03.225908 10024 raft_consensus.cc:359] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } }
I20251026 21:07:03.226116 10024 raft_consensus.cc:385] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:03.226187 10024 raft_consensus.cc:740] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438bcb34224ed783aa705830acc7bf, State: Initialized, Role: FOLLOWER
I20251026 21:07:03.226624 10024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } }
I20251026 21:07:03.226756 10024 raft_consensus.cc:399] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:03.226830 10024 raft_consensus.cc:493] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:03.226934 10024 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:03.228076 10024 raft_consensus.cc:515] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } }
I20251026 21:07:03.228432 10024 leader_election.cc:304] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86438bcb34224ed783aa705830acc7bf; no voters:
I20251026 21:07:03.228724 10024 leader_election.cc:290] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:03.228842 10029 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:03.229040 10029 raft_consensus.cc:697] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [term 1 LEADER]: Becoming Leader. State: Replica: 86438bcb34224ed783aa705830acc7bf, State: Running, Role: LEADER
I20251026 21:07:03.229321 10029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } }
I20251026 21:07:03.229841 10024 sys_catalog.cc:565] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:03.230906 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86438bcb34224ed783aa705830acc7bf. Latest consensus state: current_term: 1 leader_uuid: "86438bcb34224ed783aa705830acc7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } } }
I20251026 21:07:03.231034 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:03.231307 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86438bcb34224ed783aa705830acc7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86438bcb34224ed783aa705830acc7bf" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 37023 } } }
I20251026 21:07:03.231401 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:03.231537 10038 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:03.234390 10038 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:03.239584 10038 catalog_manager.cc:1357] Generated new cluster ID: 1384d692a21c4b80a9acfec2f37c91f7
I20251026 21:07:03.239665 10038 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:03.253199 10038 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:03.254307 10038 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:03.264655 10038 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 86438bcb34224ed783aa705830acc7bf: Generated new TSK 0
I20251026 21:07:03.265450 10038 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:03.319356 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:37023
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:03.429836 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:03.430081 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:03.430148 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:03.433583 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:03.433658 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:03.433740 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:03.438232 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:03.439465 10052 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:03.440344 10052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:03.447021 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.447021 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.447129 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:03.447198 10052 server_base.cc:1047] running on GCE node
I20251026 21:07:03.447645 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:03.448228 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:03.449376 10052 hybrid_clock.cc:648] HybridClock initialized: now 1761512823449351 us; error 39 us; skew 500 ppm
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:03.452066 10052 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:07:03.453142 10052 webserver.cc:492] Webserver started at http://127.3.175.1:36639/ using document root <none> and password file <none>
I20251026 21:07:03.453676 10052 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:03.453722 10052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:03.453928 10052 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:03.455713 10052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "73b1971f90274d8db84cfb02bfa2bb8f"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "cec6217ecc3e7e9b1d016071d4437ea6"
server_key_iv: "59e100d3499425a9af7b7c81935bab0e"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.456183 10052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "73b1971f90274d8db84cfb02bfa2bb8f"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "cec6217ecc3e7e9b1d016071d4437ea6"
server_key_iv: "59e100d3499425a9af7b7c81935bab0e"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.459751 10052 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251026 21:07:03.462090 10067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.463130 10052 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:03.463231 10052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "73b1971f90274d8db84cfb02bfa2bb8f"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "cec6217ecc3e7e9b1d016071d4437ea6"
server_key_iv: "59e100d3499425a9af7b7c81935bab0e"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.463366 10052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:03.483255 10052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:03.486323 10052 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:03.486508 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:03.487100 10052 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:03.488065 10052 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:03.488116 10052 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.488175 10052 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:03.488204 10052 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.497243 10052 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:40661
I20251026 21:07:03.497280 10180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:40661 every 8 connection(s)
I20251026 21:07:03.498286 10052 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:03.505724 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10052
I20251026 21:07:03.505869 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:03.506155 3772 external_mini_cluster.cc:1468] Setting key e4ec0b54e61454b1372b4a5bfe69548c
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:03.511659 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.500164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:34913 (local address 127.3.175.62:37023)
1026 21:07:03.500325 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:03.500330 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:03.501080 (+ 750us) server_negotiation.cc:408] Connection header received
1026 21:07:03.501940 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:03.501943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:03.501994 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:03.502083 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:03.503864 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.504382 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.505006 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.505154 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.507828 (+ 2674us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:03.507850 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:03.507853 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:03.507884 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:03.509613 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.510139 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.510143 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.510145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.510197 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.510494 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.510497 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.510499 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.510666 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:03.510758 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:03.511335 (+ 577us) server_negotiation.cc:300] Negotiation successful
1026 21:07:03.511482 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:07:03.512392 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.500447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:37023 (local address 127.3.175.1:34913)
1026 21:07:03.500905 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:03.500950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:03.501748 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:03.502240 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:03.502249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:03.502657 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:03.503677 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.503693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.504485 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.504488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:03.504897 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.504904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.505087 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.505735 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:03.505776 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:03.507640 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:03.509738 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.509745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.509759 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.510029 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.510303 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.510306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.510308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.510406 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.510773 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:03.510782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:03.511035 (+ 253us) client_negotiation.cc:769] Sending connection context
1026 21:07:03.511262 (+ 227us) client_negotiation.cc:240] Negotiation successful
1026 21:07:03.511581 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20251026 21:07:03.513985 10181 heartbeater.cc:344] Connected to a master server at 127.3.175.62:37023
I20251026 21:07:03.514277 10181 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:03.514902 10181 heartbeater.cc:507] Master 127.3.175.62:37023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:07:03.516618 9988 ts_manager.cc:194] Registered new tserver with Master: 73b1971f90274d8db84cfb02bfa2bb8f (127.3.175.1:40661)
I20251026 21:07:03.517920 9988 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:34913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:03.558449 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:37023
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:03.675623 10188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:03.675881 10188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:03.675945 10188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:03.679669 10188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:03.679747 10188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:03.679837 10188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:03.684288 10188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:03.685443 10188 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:03.686301 10188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:03.693082 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.693094 10196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.693133 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:03.693440 10188 server_base.cc:1047] running on GCE node
I20251026 21:07:03.693802 10188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:03.694407 10188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:03.695554 10188 hybrid_clock.cc:648] HybridClock initialized: now 1761512823695541 us; error 34 us; skew 500 ppm
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:03.698285 10188 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:07:03.699441 10188 webserver.cc:492] Webserver started at http://127.3.175.2:39395/ using document root <none> and password file <none>
I20251026 21:07:03.700016 10188 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:03.700073 10188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:03.700268 10188 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:03.702092 10188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "11123ca670234458b083c7a8d49b43f4"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "a70a7af23990bb51b29b6c8cd16c546a"
server_key_iv: "c87b9c3e045be4c19418be070bfbc184"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.702603 10188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "11123ca670234458b083c7a8d49b43f4"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "a70a7af23990bb51b29b6c8cd16c546a"
server_key_iv: "c87b9c3e045be4c19418be070bfbc184"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.706305 10188 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:07:03.708705 10203 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.709725 10188 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:03.709841 10188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "11123ca670234458b083c7a8d49b43f4"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "a70a7af23990bb51b29b6c8cd16c546a"
server_key_iv: "c87b9c3e045be4c19418be070bfbc184"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.709942 10188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:03.729844 10188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:03.733307 10188 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:03.733496 10188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:03.734067 10188 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:03.734998 10188 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:03.735047 10188 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.735100 10188 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:03.735131 10188 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.744136 10188 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:36753
I20251026 21:07:03.744161 10316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:36753 every 8 connection(s)
I20251026 21:07:03.745138 10188 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:03.752517 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10188
I20251026 21:07:03.752662 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:03.752946 3772 external_mini_cluster.cc:1468] Setting key 8d2050d813ba917b98b146a6fb467e40
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:03.758117 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.746885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:54071 (local address 127.3.175.62:37023)
1026 21:07:03.747037 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:03.747041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:03.747790 (+ 749us) server_negotiation.cc:408] Connection header received
1026 21:07:03.748640 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:03.748643 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:03.748704 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:03.748784 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:03.750566 (+ 1782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.751070 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.751785 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.751960 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.754558 (+ 2598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:03.754574 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:03.754577 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:03.754619 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:03.756122 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.756608 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.756612 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.756614 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.756659 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.756929 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.756933 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.756935 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.757085 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:03.757183 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:03.757755 (+ 572us) server_negotiation.cc:300] Negotiation successful
1026 21:07:03.757904 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:07:03.758620 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.747193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:37023 (local address 127.3.175.2:54071)
1026 21:07:03.747658 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:03.747693 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:03.748445 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:03.748963 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:03.748971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:03.749467 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:03.750409 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.750424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.751201 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.751203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:03.751660 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.751669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.751911 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.752619 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:03.752659 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:03.754406 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:03.756244 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.756250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.756263 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.756507 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.756742 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.756747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.756749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.756837 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.757193 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:03.757202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:03.757448 (+ 246us) client_negotiation.cc:769] Sending connection context
1026 21:07:03.757672 (+ 224us) client_negotiation.cc:240] Negotiation successful
1026 21:07:03.757919 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20251026 21:07:03.759828 10317 heartbeater.cc:344] Connected to a master server at 127.3.175.62:37023
I20251026 21:07:03.760087 10317 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:03.760610 10317 heartbeater.cc:507] Master 127.3.175.62:37023 requested a full tablet report, sending...
I20251026 21:07:03.761669 9988 ts_manager.cc:194] Registered new tserver with Master: 11123ca670234458b083c7a8d49b43f4 (127.3.175.2:36753)
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:07:03.762243 9988 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:54071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:03.800706 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:37023
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:03.912827 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:03.913090 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:03.913153 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:03.916680 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:03.916750 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:03.916857 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:03.921300 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:03.922444 10324 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:03.923398 10324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:03.930289 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.930289 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:03.930472 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:03.930835 10324 server_base.cc:1047] running on GCE node
I20251026 21:07:03.931416 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:03.932078 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:03.933279 10324 hybrid_clock.cc:648] HybridClock initialized: now 1761512823933257 us; error 38 us; skew 500 ppm
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:03.936410 10324 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:07:03.937498 10324 webserver.cc:492] Webserver started at http://127.3.175.3:37531/ using document root <none> and password file <none>
I20251026 21:07:03.938063 10324 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:03.938115 10324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:03.938300 10324 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:03.940073 10324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "f1005e9b458346d380e7ff032d59e343"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "0613960b7e3ea67b8991fe4b4b4aafb3"
server_key_iv: "4821dbfc1519937b00bc6f2f9009b1f9"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.940534 10324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "f1005e9b458346d380e7ff032d59e343"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "0613960b7e3ea67b8991fe4b4b4aafb3"
server_key_iv: "4821dbfc1519937b00bc6f2f9009b1f9"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.944022 10324 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251026 21:07:03.946326 10339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.947391 10324 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:03.947503 10324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "f1005e9b458346d380e7ff032d59e343"
format_stamp: "Formatted at 2025-10-26 21:07:03 on dist-test-slave-th6x"
server_key: "0613960b7e3ea67b8991fe4b4b4aafb3"
server_key_iv: "4821dbfc1519937b00bc6f2f9009b1f9"
server_key_version: "encryptionkey@0"
I20251026 21:07:03.947599 10324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:03.968562 10324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:03.971589 10324 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:03.971792 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:03.972397 10324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:03.973351 10324 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:03.973417 10324 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.973484 10324 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:03.973520 10324 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:03.982592 10324 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:45055
I20251026 21:07:03.982616 10452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:45055 every 8 connection(s)
I20251026 21:07:03.983641 10324 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:03.987128 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10324
I20251026 21:07:03.987268 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:03.987562 3772 external_mini_cluster.cc:1468] Setting key 2c39bc2154148c51a3bbd46161608599
Oct 26 21:07:03 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:03.996466 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.985436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:57885 (local address 127.3.175.62:37023)
1026 21:07:03.985626 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:03.985631 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:03.986356 (+ 725us) server_negotiation.cc:408] Connection header received
1026 21:07:03.987356 (+ 1000us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:03.987360 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:03.987426 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:03.987505 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:03.989592 (+ 2087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.990131 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.990882 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.991035 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.993213 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:03.993233 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:03.993235 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:03.993266 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:03.994617 (+ 1351us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.995187 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.995190 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.995192 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.995233 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:03.995496 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:03.995498 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:03.995499 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:03.995651 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:03.995729 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:03.996215 (+ 486us) server_negotiation.cc:300] Negotiation successful
1026 21:07:03.996326 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251026 21:07:03.997164 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:03.985745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:37023 (local address 127.3.175.3:57885)
1026 21:07:03.986197 (+ 452us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:03.986235 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:03.987067 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:03.987721 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:03.987733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:03.988273 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:03.989420 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.989439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.990238 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:03.990240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:03.990659 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:03.990666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:03.990851 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:03.991528 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:03.991555 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:03.993055 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:03.994730 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.994735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.994745 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.995093 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.995329 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:03.995334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:03.995336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:03.995426 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:03.995739 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:03.995744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:03.995993 (+ 249us) client_negotiation.cc:769] Sending connection context
1026 21:07:03.996172 (+ 179us) client_negotiation.cc:240] Negotiation successful
1026 21:07:03.996405 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20251026 21:07:03.998342 10453 heartbeater.cc:344] Connected to a master server at 127.3.175.62:37023
I20251026 21:07:03.998592 10453 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:03.999138 10453 heartbeater.cc:507] Master 127.3.175.62:37023 requested a full tablet report, sending...
I20251026 21:07:04.000334 9988 ts_manager.cc:194] Registered new tserver with Master: f1005e9b458346d380e7ff032d59e343 (127.3.175.3:45055)
I20251026 21:07:04.001107 9988 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:57885
I20251026 21:07:04.001478 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/test-admin.keytab.
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.016420 3772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251026 21:07:04.019050 10460 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:04.017846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.3.175.1:40661)
1026 21:07:04.018143 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.018149 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.018171 (+ 22us) server_negotiation.cc:408] Connection header received
1026 21:07:04.018222 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.018227 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.018389 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.018503 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.018899 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53392: BlockingRecv error: recv got EOF from 127.0.0.1:53392 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20251026 21:07:04.019315 3772 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc)
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.020342 3772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.021253 3772 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:07:04.030524 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:04.022194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.3.175.1:40661)
1026 21:07:04.022330 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.022337 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.022354 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:07:04.022496 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.022499 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.022544 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.022615 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.023347 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.024186 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:04.024860 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.025138 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:04.027205 (+ 2067us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:04.027249 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:04.027262 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:04.027362 (+ 100us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:04.028952 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.029360 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.029366 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.029368 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.029423 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.029643 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.029649 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.029651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.029843 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:04.030086 (+ 243us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:04.030230 (+ 144us) server_negotiation.cc:300] Negotiation successful
1026 21:07:04.030378 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20251026 21:07:04.034085 10460 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:04.033292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.3.175.1:40661)
1026 21:07:04.033428 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.033431 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.033443 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:04.033545 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.033547 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.033584 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.033649 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.033974 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53418: BlockingRecv error: recv got EOF from 127.0.0.1:53418 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20251026 21:07:04.034170 3772 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc)
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.035183 3772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.036021 3772 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:07:04.044355 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:04.036893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.3.175.1:40661)
1026 21:07:04.037032 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.037039 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.037054 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:04.037213 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.037216 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.037262 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.037332 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.037941 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.038411 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:04.039000 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.039210 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:04.041349 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:04.041371 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:04.041373 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:04.041399 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:04.042795 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.043258 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.043261 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.043263 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.043333 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.043551 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.043554 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.043556 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.043726 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:04.043856 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:04.044069 (+ 213us) server_negotiation.cc:300] Negotiation successful
1026 21:07:04.044186 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20251026 21:07:04.047058 10460 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:04.046146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.3.175.1:40661)
1026 21:07:04.046269 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.046274 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.046336 (+ 62us) server_negotiation.cc:408] Connection header received
1026 21:07:04.046458 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.046461 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.046508 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.046592 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.046962 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53430: BlockingRecv error: recv got EOF from 127.0.0.1:53430 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20251026 21:07:04.047195 3772 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761512762855912-3772-0/krb5kdc/krb5cc' of type FILE
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.048306 3772 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:04.049180 3772 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.1@KRBTEST.COM
I20251026 21:07:04.057363 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:04.050077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.3.175.1:40661)
1026 21:07:04.050189 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:04.050200 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:04.050213 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:04.050252 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:04.050254 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:04.050290 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:04.050363 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:04.050926 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.051478 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:04.052092 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:04.052268 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:04.054332 (+ 2064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:04.054357 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:04.054359 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:04.054386 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:04.056109 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.056455 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.056458 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.056459 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.056495 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:04.056650 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:04.056652 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:04.056653 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:04.056791 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:04.056888 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:04.057130 (+ 242us) server_negotiation.cc:300] Negotiation successful
1026 21:07:04.057232 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251026 21:07:04.058450 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10052
I20251026 21:07:04.064235 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10188
I20251026 21:07:04.070109 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10324
I20251026 21:07:04.075706 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 9956
2025-10-26T21:07:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3204 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[10473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[10473](info): set up 2 sockets
Oct 26 21:07:04 dist-test-slave-th6x krb5kdc[10473](info): commencing operation
krb5kdc: starting...
W20251026 21:07:06.072295 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:06.214464 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:41917
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:06.323129 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:06.323432 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:06.323493 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:06.326805 10489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:06.326890 10489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:06.326915 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:06.326933 10489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:06.326951 10489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:06.331549 10489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:06.332654 10489 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:06.333540 10489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:06.339308 10495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:06.339359 10497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:06.339365 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:06.339483 10489 server_base.cc:1047] running on GCE node
I20251026 21:07:06.340345 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:06.341347 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:06.342499 10489 hybrid_clock.cc:648] HybridClock initialized: now 1761512826342484 us; error 32 us; skew 500 ppm
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:06.345467 10489 init.cc:377] Logged in from keytab as oryx/127.3.175.62@KRBTEST.COM (short username oryx)
I20251026 21:07:06.346644 10489 webserver.cc:492] Webserver started at http://127.3.175.62:40957/ using document root <none> and password file <none>
I20251026 21:07:06.347231 10489 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:06.347319 10489 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:06.347527 10489 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:06.349311 10489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "1c2e62502e21436a81044f938290c8f2"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "530d5c56d12d6e36b3e6df3eae09955a"
server_key_iv: "484587224a419a7fecd8af373fb0db42"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.349815 10489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "1c2e62502e21436a81044f938290c8f2"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "530d5c56d12d6e36b3e6df3eae09955a"
server_key_iv: "484587224a419a7fecd8af373fb0db42"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.353396 10489 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251026 21:07:06.355703 10504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.356902 10489 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:06.357060 10489 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "1c2e62502e21436a81044f938290c8f2"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "530d5c56d12d6e36b3e6df3eae09955a"
server_key_iv: "484587224a419a7fecd8af373fb0db42"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.357174 10489 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:06.373431 10489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:06.376708 10489 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:06.376986 10489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:06.384549 10489 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:41917
I20251026 21:07:06.384554 10556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:41917 every 8 connection(s)
I20251026 21:07:06.385641 10489 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:06.388569 10557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:06.390692 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10489
I20251026 21:07:06.390825 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:06.391059 3772 external_mini_cluster.cc:1468] Setting key 7927767cfb07441c99ccf5148423bf70
I20251026 21:07:06.395411 10557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Bootstrap starting.
I20251026 21:07:06.397989 10557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:06.398770 10557 log.cc:826] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Log is configured to *not* fsync() on all Append() calls
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:06.400890 10557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: No bootstrap required, opened a new log
I20251026 21:07:06.403945 10557 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } }
I20251026 21:07:06.404172 10557 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:06.404223 10557 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2e62502e21436a81044f938290c8f2, State: Initialized, Role: FOLLOWER
I20251026 21:07:06.404489 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:06.392463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.3.175.62:41917)
1026 21:07:06.392906 (+ 443us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:06.392918 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:06.392959 (+ 41us) server_negotiation.cc:408] Connection header received
1026 21:07:06.393608 (+ 649us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:06.393636 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:06.394046 (+ 410us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:06.394423 (+ 377us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:06.395232 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.396408 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:06.397074 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.397354 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:06.399945 (+ 2591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:06.399966 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:06.399979 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:06.400012 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:06.401870 (+ 1858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.402354 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.402359 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.402363 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.402438 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.402657 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.402659 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.402660 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.403004 (+ 344us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:06.403183 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:06.403470 (+ 287us) server_negotiation.cc:300] Negotiation successful
1026 21:07:06.403749 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20251026 21:07:06.404690 10557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } }
I20251026 21:07:06.404832 10557 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:06.404891 10557 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:06.404992 10557 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:06.405977 10557 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } }
I20251026 21:07:06.406368 10557 leader_election.cc:304] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c2e62502e21436a81044f938290c8f2; no voters:
I20251026 21:07:06.406692 10557 leader_election.cc:290] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:06.406783 10562 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:06.406941 10562 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2e62502e21436a81044f938290c8f2, State: Running, Role: LEADER
I20251026 21:07:06.407233 10562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } }
I20251026 21:07:06.407941 10557 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:06.409317 10563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } }
I20251026 21:07:06.409349 10564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c2e62502e21436a81044f938290c8f2. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } }
I20251026 21:07:06.409435 10563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:06.409469 10564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:06.410368 10566 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:06.413478 10566 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:06.418408 10566 catalog_manager.cc:1357] Generated new cluster ID: a744a34c9b134fdc866f41e652cc9d9b
I20251026 21:07:06.418486 10566 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:06.427557 10566 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:06.428771 10566 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:06.435981 10566 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Generated new TSK 0
I20251026 21:07:06.436708 10566 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:06.491154 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:41917
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:06.602358 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:06.602595 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:06.602660 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:06.606112 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:06.606196 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:06.606302 10585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:06.610834 10585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:06.612066 10585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:06.612970 10585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:06.619678 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:06.619868 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:06.619760 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:06.619838 10585 server_base.cc:1047] running on GCE node
I20251026 21:07:06.620430 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:06.620975 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:06.622131 10585 hybrid_clock.cc:648] HybridClock initialized: now 1761512826622119 us; error 27 us; skew 500 ppm
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:06.625063 10585 init.cc:377] Logged in from keytab as oryx/127.3.175.1@KRBTEST.COM (short username oryx)
I20251026 21:07:06.626164 10585 webserver.cc:492] Webserver started at http://127.3.175.1:46579/ using document root <none> and password file <none>
I20251026 21:07:06.626745 10585 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:06.626804 10585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:06.626991 10585 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:06.628729 10585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "643943dcd2704d72852a37536e03a82f"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "972ffb961053daa646fcf7ff42e0baa2"
server_key_iv: "8710288f09ec97715cfeec0457a957e3"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.629215 10585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "643943dcd2704d72852a37536e03a82f"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "972ffb961053daa646fcf7ff42e0baa2"
server_key_iv: "8710288f09ec97715cfeec0457a957e3"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.634056 10585 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251026 21:07:06.637148 10600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.638402 10585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:06.638513 10585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "643943dcd2704d72852a37536e03a82f"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "972ffb961053daa646fcf7ff42e0baa2"
server_key_iv: "8710288f09ec97715cfeec0457a957e3"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.638625 10585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:06.663028 10585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:06.666172 10585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:06.666340 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:06.666848 10585 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:06.667762 10585 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:06.667811 10585 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.667852 10585 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:06.667882 10585 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.676811 10585 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:40987
I20251026 21:07:06.676885 10713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:40987 every 8 connection(s)
I20251026 21:07:06.677788 10585 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:06.687601 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10585
I20251026 21:07:06.687705 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:06.687939 3772 external_mini_cluster.cc:1468] Setting key bd05d1bc3a79f08c6cd6ddd568ca9088
I20251026 21:07:06.691010 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:06.679626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:39971 (local address 127.3.175.62:41917)
1026 21:07:06.679815 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:06.679819 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:06.680531 (+ 712us) server_negotiation.cc:408] Connection header received
1026 21:07:06.681381 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:06.681385 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:06.681443 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:06.681527 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:06.682800 (+ 1273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.683355 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:06.683986 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.684171 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:06.686942 (+ 2771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:06.686966 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:06.686969 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:06.686997 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:06.688628 (+ 1631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.689294 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.689296 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.689298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.689354 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.689691 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.689694 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.689696 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.689901 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:06.690002 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:06.690564 (+ 562us) server_negotiation.cc:300] Negotiation successful
1026 21:07:06.690734 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:07:06.691622 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:06.679934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:39971)
1026 21:07:06.680395 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:06.680430 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:06.681190 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:06.681691 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:06.681700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:06.682104 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:06.682652 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:06.682664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.683469 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:06.683473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:06.683871 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:06.683878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.684100 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:06.685235 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:06.685258 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:06.686769 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:06.688789 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:06.688796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:06.688810 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:06.689194 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:06.689448 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:06.689451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:06.689453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:06.689592 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:06.690010 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:06.690016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:06.690286 (+ 270us) client_negotiation.cc:769] Sending connection context
1026 21:07:06.690496 (+ 210us) client_negotiation.cc:240] Negotiation successful
1026 21:07:06.690747 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
I20251026 21:07:06.692816 10714 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:06.693069 10714 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:06.693608 10714 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:06.695117 10521 ts_manager.cc:194] Registered new tserver with Master: 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
I20251026 21:07:06.696334 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:39971
WARNING: no policy specified for oryx/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:06.738909 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:41917
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:06.850593 10721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:06.850875 10721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:06.850936 10721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:06.854487 10721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:06.854573 10721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:06.854660 10721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:06.859412 10721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:06.860574 10721 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:06.861464 10721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:06.868747 10727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:06.868747 10729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:06.868983 10721 server_base.cc:1047] running on GCE node
W20251026 21:07:06.869014 10726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:06.869553 10721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:06.870231 10721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:06.871384 10721 hybrid_clock.cc:648] HybridClock initialized: now 1761512826871355 us; error 38 us; skew 500 ppm
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:06.874526 10721 init.cc:377] Logged in from keytab as oryx/127.3.175.2@KRBTEST.COM (short username oryx)
I20251026 21:07:06.875801 10721 webserver.cc:492] Webserver started at http://127.3.175.2:37501/ using document root <none> and password file <none>
I20251026 21:07:06.876389 10721 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:06.876453 10721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:06.876650 10721 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:06.878544 10721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "226478220f964d02baead480f14a9c72"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "75b06b237124bd69f2fe90af37a2c64b"
server_key_iv: "00c30211a720e57bb80501da8589a53f"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.879072 10721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "226478220f964d02baead480f14a9c72"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "75b06b237124bd69f2fe90af37a2c64b"
server_key_iv: "00c30211a720e57bb80501da8589a53f"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.882767 10721 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:06.885247 10736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.886345 10721 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:06.886472 10721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "226478220f964d02baead480f14a9c72"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "75b06b237124bd69f2fe90af37a2c64b"
server_key_iv: "00c30211a720e57bb80501da8589a53f"
server_key_version: "encryptionkey@0"
I20251026 21:07:06.886586 10721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:06.906744 10721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:06.910220 10721 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:06.910424 10721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:06.911018 10721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:06.911998 10721 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:06.912052 10721 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.912113 10721 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:06.912149 10721 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:06.921005 10721 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:41823
I20251026 21:07:06.921042 10849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:41823 every 8 connection(s)
I20251026 21:07:06.922000 10721 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:06.925273 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10721
I20251026 21:07:06.925372 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:06.925643 3772 external_mini_cluster.cc:1468] Setting key 5f9a41095b0e9743d8d4ba851d88ec61
Oct 26 21:07:06 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:07:06.936043 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:06.923792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:48371 (local address 127.3.175.62:41917)
1026 21:07:06.923958 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:06.923962 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:06.924644 (+ 682us) server_negotiation.cc:408] Connection header received
1026 21:07:06.925512 (+ 868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:06.925517 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:06.925583 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:06.925684 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:06.927190 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.927982 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:06.928737 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.928922 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:06.931981 (+ 3059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:06.932002 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:06.932005 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:06.932034 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:06.933710 (+ 1676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.934393 (+ 683us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.934397 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.934398 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.934451 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:06.934778 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:06.934780 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:06.934781 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:06.934948 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:06.935085 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:06.935732 (+ 647us) server_negotiation.cc:300] Negotiation successful
1026 21:07:06.935881 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251026 21:07:06.936765 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:06.924055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:48371)
1026 21:07:06.924487 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:06.924525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:06.925300 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:06.925863 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:06.925872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:06.926378 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:06.927036 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:06.927051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.928123 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:06.928126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:06.928536 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:06.928548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:06.928862 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:06.930057 (+ 1195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:06.930083 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:06.931790 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:06.933853 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:06.933860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:06.933877 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:06.934261 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:06.934554 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:06.934558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:06.934563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:06.934692 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:06.935072 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:06.935081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:06.935459 (+ 378us) client_negotiation.cc:769] Sending connection context
1026 21:07:06.935669 (+ 210us) client_negotiation.cc:240] Negotiation successful
1026 21:07:06.935995 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.3@KRBTEST.COM" created.
I20251026 21:07:06.938036 10850 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:06.938293 10850 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:06.938830 10850 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:06.939913 10521 ts_manager.cc:194] Registered new tserver with Master: 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:06.940721 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:48371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:06.976048 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:41917
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:07.087733 10857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:07.088001 10857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:07.088063 10857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:07.091645 10857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:07.091727 10857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:07.091871 10857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:07.097877 10857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:07.099144 10857 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:07.100097 10857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:07.107209 10865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.107360 10857 server_base.cc:1047] running on GCE node
W20251026 21:07:07.107445 10863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.107214 10862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.107897 10857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:07.108518 10857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:07.109678 10857 hybrid_clock.cc:648] HybridClock initialized: now 1761512827109664 us; error 35 us; skew 500 ppm
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:07.112792 10857 init.cc:377] Logged in from keytab as oryx/127.3.175.3@KRBTEST.COM (short username oryx)
I20251026 21:07:07.113943 10857 webserver.cc:492] Webserver started at http://127.3.175.3:33487/ using document root <none> and password file <none>
I20251026 21:07:07.114519 10857 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.114576 10857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.114773 10857 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:07.116551 10857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "8b0950373eb54a7886e1314b21421055"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "bb4f3d30df36b76a0fc1d3caae527c22"
server_key_iv: "11a364b082cef1836dbab89c0256bddb"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.117041 10857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "8b0950373eb54a7886e1314b21421055"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "bb4f3d30df36b76a0fc1d3caae527c22"
server_key_iv: "11a364b082cef1836dbab89c0256bddb"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.120487 10857 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:07.122761 10872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.123898 10857 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:07.124037 10857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "8b0950373eb54a7886e1314b21421055"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "bb4f3d30df36b76a0fc1d3caae527c22"
server_key_iv: "11a364b082cef1836dbab89c0256bddb"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.124156 10857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.140889 10857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:07.144292 10857 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:07.144510 10857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:07.145119 10857 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:07.146127 10857 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:07.146185 10857 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.146255 10857 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:07.146286 10857 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.156533 10857 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:42287
I20251026 21:07:07.156546 10985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:42287 every 8 connection(s)
I20251026 21:07:07.157593 10857 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:07.162353 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10857
I20251026 21:07:07.162461 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:07.162684 3772 external_mini_cluster.cc:1468] Setting key 9165171af51c9d4025ebf9e084785608
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:07.170658 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.159439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:39375 (local address 127.3.175.62:41917)
1026 21:07:07.159607 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.159612 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.160368 (+ 756us) server_negotiation.cc:408] Connection header received
1026 21:07:07.161272 (+ 904us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.161276 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.161330 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.161435 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.163003 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.163563 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.164232 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.164415 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.167217 (+ 2802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.167236 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.167238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.167268 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.168680 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.169206 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.169209 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.169211 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.169259 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.169532 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.169536 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.169538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.169703 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.169799 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.170340 (+ 541us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.170481 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:07:07.171394 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.159712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:39375)
1026 21:07:07.160210 (+ 498us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.160251 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.161034 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.161682 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.161692 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.162165 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.162823 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.162838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.163701 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.163706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.164117 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.164124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.164360 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.165491 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.165517 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.167034 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.168817 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.168824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.168840 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.169094 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.169344 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.169347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.169349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.169441 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.169809 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.169817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.170078 (+ 261us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.170298 (+ 220us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.170592 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20251026 21:07:07.172435 10986 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:07.172704 10986 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:07.173276 10986 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:07.174327 10521 ts_manager.cc:194] Registered new tserver with Master: 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287)
I20251026 21:07:07.174902 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:39375
I20251026 21:07:07.176466 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.175.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:07.227320 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.61:34739
--webserver_interface=127.3.175.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:41917,127.3.175.61:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:07.338600 10993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:07.338868 10993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:07.338925 10993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:07.342293 10993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:07.342370 10993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:07.342394 10993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:07.342417 10993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:07.342454 10993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:07.346932 10993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:41917,127.3.175.61:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.61:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.175.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
W20251026 21:07:07.347179 10993 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.175.62:41917,127.3.175.61:34739'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251026 21:07:07.348196 10993 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:07.349076 10993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:07.355129 10999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.355312 10993 server_base.cc:1047] running on GCE node
W20251026 21:07:07.355150 11001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.355160 10998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.356017 10993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:07.356911 10993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:07.358117 10993 hybrid_clock.cc:648] HybridClock initialized: now 1761512827358103 us; error 33 us; skew 500 ppm
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:07.361238 10993 init.cc:377] Logged in from keytab as oryx/127.3.175.61@KRBTEST.COM (short username oryx)
I20251026 21:07:07.362452 10993 webserver.cc:492] Webserver started at http://127.3.175.61:39265/ using document root <none> and password file <none>
I20251026 21:07:07.363101 10993 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.363159 10993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.363405 10993 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:07.365186 10993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/instance:
uuid: "02b06b2ce5944ee9b05df06891a13b35"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "704e6856c7bfa244a68ca53238b67a38"
server_key_iv: "d6140a812097fc2f0f1c28fe6595312a"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.365681 10993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal/instance:
uuid: "02b06b2ce5944ee9b05df06891a13b35"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "704e6856c7bfa244a68ca53238b67a38"
server_key_iv: "d6140a812097fc2f0f1c28fe6595312a"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.369489 10993 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251026 21:07:07.372014 11008 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.373129 10993 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:07.373250 10993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
uuid: "02b06b2ce5944ee9b05df06891a13b35"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "704e6856c7bfa244a68ca53238b67a38"
server_key_iv: "d6140a812097fc2f0f1c28fe6595312a"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.373354 10993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.383962 10993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:07.387353 10993 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:07.387547 10993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:07.394646 10993 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.61:34739
I20251026 21:07:07.394845 11060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.61:34739 every 8 connection(s)
I20251026 21:07:07.395746 10993 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/info.pb
I20251026 21:07:07.398926 11061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:07.402912 11061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } has no permanent_uuid. Determining permanent_uuid...
I20251026 21:07:07.403525 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 10993
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.61@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:07.416107 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.404855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.3.175.62:41917)
1026 21:07:07.404990 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.404993 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.405486 (+ 493us) server_negotiation.cc:408] Connection header received
1026 21:07:07.406205 (+ 719us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.406209 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.406264 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.406361 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.407584 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.408376 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.409229 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.409402 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.412075 (+ 2673us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.412099 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.412102 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.412141 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.413971 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.414515 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.414522 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.414526 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.414583 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.414893 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.414896 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.414897 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.415094 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.415194 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.415783 (+ 589us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.415928 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:07:07.416831 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.404861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.3.175.61:34739)
1026 21:07:07.405766 (+ 905us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.405771 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.405808 (+ 37us) server_negotiation.cc:408] Connection header received
1026 21:07:07.406024 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.406042 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.406238 (+ 196us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.406405 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.407476 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.408559 (+ 1083us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.409399 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.409607 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.412460 (+ 2853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.412478 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.412489 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.412511 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.414650 (+ 2139us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.415152 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.415161 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.415165 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.415222 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.415470 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.415472 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.415473 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.415647 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.415836 (+ 189us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.416085 (+ 249us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.416457 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"spinlock_wait_cycles":59776,"thread_start_us":156,"threads_started":1}
I20251026 21:07:07.416836 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.404860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49880)
1026 21:07:07.405349 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.405386 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.406013 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.406414 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.406420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.406951 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.407429 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.407446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.408501 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.408503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.409088 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.409097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.409317 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.410053 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.410090 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.411925 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.414128 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.414137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.414148 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.414410 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.414676 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.414679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.414680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.414790 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.415200 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.415206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.415537 (+ 331us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.415727 (+ 190us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.416030 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":156,"threads_started":1}
I20251026 21:07:07.418244 11061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.61@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:07.425397 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.418730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58010)
1026 21:07:07.418826 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.418843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.418918 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.419170 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.419172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.419369 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.419531 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.419537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.420288 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.420290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.420625 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.420631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.420721 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.421219 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.421231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.422437 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.424211 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.424214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.424216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.424369 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.424584 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.424587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.424588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.424645 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.424931 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.424933 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.425028 (+ 95us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.425125 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.425238 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20251026 21:07:07.425420 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.418893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.3.175.61:34739)
1026 21:07:07.418999 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.419002 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.419013 (+ 11us) server_negotiation.cc:408] Connection header received
1026 21:07:07.419039 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.419041 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.419075 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.419147 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.419650 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.420188 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.420709 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.420840 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.422551 (+ 1711us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.422566 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.422568 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.422589 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.424106 (+ 1517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.424446 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.424448 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.424450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.424482 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.424724 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.424726 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.424727 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.424859 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.424925 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.425175 (+ 250us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.425294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":39}
I20251026 21:07:07.428278 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Bootstrap starting.
I20251026 21:07:07.430816 11061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:07.431567 11061 log.cc:826] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:07.433385 11061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: No bootstrap required, opened a new log
I20251026 21:07:07.435989 11061 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } }
I20251026 21:07:07.436179 11061 raft_consensus.cc:385] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:07.436218 11061 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b06b2ce5944ee9b05df06891a13b35, State: Initialized, Role: FOLLOWER
I20251026 21:07:07.436666 11061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } }
I20251026 21:07:07.437130 11067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } } }
I20251026 21:07:07.437247 11067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:07.437820 11061 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 21:07:07.442394 11078 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 21:07:07.442502 11078 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 21:07:07.444697 10993 master_runner.cc:428] Detected that this master 02b06b2ce5944ee9b05df06891a13b35 is joining an existing cluster
I20251026 21:07:07.444784 10993 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.175.61:34739
I20251026 21:07:07.454325 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.447334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.3.175.62:41917)
1026 21:07:07.447466 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.447470 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.447752 (+ 282us) server_negotiation.cc:408] Connection header received
1026 21:07:07.447906 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.447908 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.447955 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.448025 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.448671 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.449438 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.450140 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.450348 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.451312 (+ 964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.451335 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.451338 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.451365 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.452884 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.453278 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.453282 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.453283 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.453328 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.453540 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.453543 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.453545 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.453697 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.453788 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.454042 (+ 254us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.454160 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251026 21:07:07.454345 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.447314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49896)
1026 21:07:07.447659 (+ 345us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.447671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.447772 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.448064 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.448067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.448301 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.448548 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.448555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.449596 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.449598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.450031 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.450038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.450147 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.450784 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.450797 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.451152 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.452997 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.453000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.453003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.453180 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.453413 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.453417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.453418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.453462 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.453786 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.453789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.453881 (+ 92us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.453982 (+ 101us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.454135 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":81,"threads_started":1}
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.62@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:07.476640 11067 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:07.476876 11067 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } }
I20251026 21:07:07.478001 11067 leader_election.cc:290] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917)
I20251026 21:07:07.479223 10531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02b06b2ce5944ee9b05df06891a13b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2e62502e21436a81044f938290c8f2" is_pre_election: true
WARNING: no policy specified for oryx/127.3.175.60@KRBTEST.COM; defaulting to no policy
I20251026 21:07:07.480365 11012 leader_election.cc:304] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35; no voters: 1c2e62502e21436a81044f938290c8f2
I20251026 21:07:07.480865 11067 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.60@KRBTEST.COM" created.
I20251026 21:07:07.482434 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.447214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.3.175.61:34739)
1026 21:07:07.447361 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.447365 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.447598 (+ 233us) server_negotiation.cc:408] Connection header received
1026 21:07:07.447754 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.447758 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.447795 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.447840 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.448794 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.449431 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.450252 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.450442 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.478505 (+ 28063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.478528 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.478531 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.478557 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.480720 (+ 2163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.481252 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.481257 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.481258 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.481304 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.481625 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.481631 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.481635 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.481790 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.481854 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.482128 (+ 274us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.482252 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 21:07:07.482503 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.458973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.3.175.61:34739)
1026 21:07:07.459482 (+ 509us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.459486 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.459499 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:07.459658 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.459661 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.459709 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.459786 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.460606 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.461125 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.461710 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.461956 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.464367 (+ 2411us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.464388 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.464390 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.464416 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.477483 (+ 13067us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.480938 (+ 3455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.480942 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.480945 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.481000 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.481298 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.481300 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.481302 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.481517 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.481613 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.482104 (+ 491us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.482346 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":434,"thread_start_us":150,"threads_started":1}
I20251026 21:07:07.482939 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.447227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58020)
1026 21:07:07.447523 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.447537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.447661 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.447967 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.447970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.448323 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.448668 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.448680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.449547 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.449551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.450130 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.450139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.450570 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.477890 (+ 27320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.477925 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.478367 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.480841 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.480844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.480846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.481150 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.481392 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.481395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.481397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.481437 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.481897 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.481899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.481952 (+ 53us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.482617 (+ 665us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.482774 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":161,"threads_started":1}
I20251026 21:07:07.484515 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.458887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58032)
1026 21:07:07.459317 (+ 430us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.459368 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.459519 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.459842 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.459847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.460264 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.460483 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.460490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.461235 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.461238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.461605 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.461612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.461725 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.462464 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.462479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.464210 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.479038 (+ 14828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.479053 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.479584 (+ 531us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.480760 (+ 1176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.481119 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.481123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.481125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.481188 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.481685 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.481695 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.481840 (+ 145us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.484119 (+ 2279us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.484342 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20251026 21:07:07.485790 10521 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.3.175.61:34739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35"
member_type: NON_VOTER
last_known_addr {
host: "127.3.175.61"
port: 34739
}
attrs {
promote: true
}
}
dest_uuid: "1c2e62502e21436a81044f938290c8f2"
cas_config_opid_index: -1
I20251026 21:07:07.486441 10521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } }
I20251026 21:07:07.488305 11091 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } }
I20251026 21:07:07.488938 11035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:07.489966 11091 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.3.175.61:34739
I20251026 21:07:07.490257 11035 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 21:07:07.490499 10993 master.cc:561] Master@127.3.175.61:34739 shutting down...
I20251026 21:07:07.491046 11091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } }
I20251026 21:07:07.491160 11091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:07.491948 11092 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 21:07:07.494333 10505 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251026 21:07:07.495240 10505 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 -> Peer 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61:34739): Couldn't send request to peer 02b06b2ce5944ee9b05df06891a13b35. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 21:07:07.496970 10993 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:07.497262 10993 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:07.497364 10993 tablet_replica.cc:333] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:07.503052 10993 master.cc:583] Master@127.3.175.61:34739 shutdown complete.
I20251026 21:07:07.503115 10993 master_runner.cc:305] Clearing existing system tablet
I20251026 21:07:07.503371 10993 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.503427 10993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.504535 10993 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.505311 11097 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.505534 10993 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.505662 10993 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.505702 10993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
uuid: "02b06b2ce5944ee9b05df06891a13b35"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "704e6856c7bfa244a68ca53238b67a38"
server_key_iv: "d6140a812097fc2f0f1c28fe6595312a"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.507509 10993 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Deleting tablet data with delete state TABLET_DATA_DELETED
WARNING: no policy specified for HTTP/127.3.175.60@KRBTEST.COM; defaulting to no policy
I20251026 21:07:07.509471 10993 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251026 21:07:07.509560 10993 log.cc:1199] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251026 21:07:07.510181 10993 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Deleting consensus metadata
I20251026 21:07:07.510340 10993 master_runner.cc:315] Copying system tablet from 127.3.175.62:41917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.60@KRBTEST.COM" created.
I20251026 21:07:07.511618 10993 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: tablet copy: Beginning tablet copy session from remote peer at address 127.3.175.62:41917
I20251026 21:07:07.520171 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.512327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.3.175.62:41917)
1026 21:07:07.512492 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.512499 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.512614 (+ 115us) server_negotiation.cc:408] Connection header received
1026 21:07:07.512809 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.512815 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.512894 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.513006 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.513689 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.514499 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.515446 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.516157 (+ 711us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.516649 (+ 492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.516675 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.516680 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.516711 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.518570 (+ 1859us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.519031 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.519035 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.519037 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.519096 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.519377 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.519380 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.519381 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.519528 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.519606 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.519936 (+ 330us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.520036 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 21:07:07.520215 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.512215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49910)
1026 21:07:07.512521 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.512536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.512698 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.513049 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.513052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.513259 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.513547 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.513555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.514624 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.514629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.515246 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.515255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.515446 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.516110 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.516128 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.516513 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.518689 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.518693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.518696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.518925 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.519198 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.519202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.519204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.519260 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.519614 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.519617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.519715 (+ 98us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.519870 (+ 155us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.520018 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20251026 21:07:07.520805 10541 tablet_copy_service.cc:140] P 1c2e62502e21436a81044f938290c8f2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02b06b2ce5944ee9b05df06891a13b35 ({username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:49910)
I20251026 21:07:07.520946 10541 tablet_copy_service.cc:161] P 1c2e62502e21436a81044f938290c8f2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02b06b2ce5944ee9b05df06891a13b35 at {username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:49910: session id = 02b06b2ce5944ee9b05df06891a13b35-00000000000000000000000000000000
I20251026 21:07:07.522284 10541 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 21:07:07.523905 10993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:07.525755 10993 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: tablet copy: Starting download of 0 data blocks...
I20251026 21:07:07.525954 10993 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: tablet copy: Starting download of 1 WAL segments...
I20251026 21:07:07.526470 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.60:36199
--webserver_interface=127.3.175.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 21:07:07.528833 10993 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 21:07:07.530540 10541 tablet_copy_service.cc:342] P 1c2e62502e21436a81044f938290c8f2: Request end of tablet copy session 02b06b2ce5944ee9b05df06891a13b35-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:49910
I20251026 21:07:07.530651 10541 tablet_copy_service.cc:434] P 1c2e62502e21436a81044f938290c8f2: ending tablet copy session 02b06b2ce5944ee9b05df06891a13b35-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02b06b2ce5944ee9b05df06891a13b35
W20251026 21:07:07.536027 10993 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251026 21:07:07.538270 10993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:07.540195 11106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.540398 11109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.540293 11107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.540570 10993 server_base.cc:1047] running on GCE node
I20251026 21:07:07.540809 10993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:07.541115 10993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:07.542248 10993 hybrid_clock.cc:648] HybridClock initialized: now 1761512827542227 us; error 34 us; skew 500 ppm
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:07.543834 10993 init.cc:377] Logged in from keytab as oryx/127.3.175.61@KRBTEST.COM (short username oryx)
I20251026 21:07:07.544320 10993 webserver.cc:492] Webserver started at http://127.3.175.61:43761/ using document root <none> and password file <none>
I20251026 21:07:07.544502 10993 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.544540 10993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.545673 10993 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.546548 11116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.546772 10993 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.546845 10993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
uuid: "02b06b2ce5944ee9b05df06891a13b35"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "704e6856c7bfa244a68ca53238b67a38"
server_key_iv: "d6140a812097fc2f0f1c28fe6595312a"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.546933 10993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.554028 10993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:07.556020 10993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:07.560789 10993 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.61:34739
I20251026 21:07:07.560798 11168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.61:34739 every 8 connection(s)
I20251026 21:07:07.561506 10993 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-1/data/info.pb
I20251026 21:07:07.562942 11169 sys_catalog.cc:263] Verifying existing consensus state
I20251026 21:07:07.563820 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Bootstrap starting.
I20251026 21:07:07.572151 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 21:07:07.572486 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Bootstrap complete.
I20251026 21:07:07.573040 11169 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } }
I20251026 21:07:07.573168 11169 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02b06b2ce5944ee9b05df06891a13b35, State: Initialized, Role: LEARNER
I20251026 21:07:07.573290 11169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } }
I20251026 21:07:07.573632 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } }
I20251026 21:07:07.573776 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: LEARNER
I20251026 21:07:07.573800 11169 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:07.577246 11183 catalog_manager.cc:1269] Loaded cluster ID: a744a34c9b134fdc866f41e652cc9d9b
I20251026 21:07:07.577307 11183 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: loading cluster ID for follower catalog manager: success
I20251026 21:07:07.579033 11183 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: acquiring CA information for follower catalog manager: success
I20251026 21:07:07.580205 11183 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251026 21:07:07.683650 11105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:07.684062 11105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:07.684144 11105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:07.689031 11105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:07.689141 11105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:07.689174 11105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:07.689216 11105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:07.689240 11105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:07.694732 11105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.60:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.175.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:07.695971 11105 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:07.696831 11105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 21:07:07.698908 10714 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
W20251026 21:07:07.702925 11189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.702943 11191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.703051 11188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.703076 11105 server_base.cc:1047] running on GCE node
I20251026 21:07:07.703794 11105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:07.704804 11105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:07.705986 11105 hybrid_clock.cc:648] HybridClock initialized: now 1761512827705973 us; error 43 us; skew 500 ppm
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:07.709112 11105 init.cc:377] Logged in from keytab as oryx/127.3.175.60@KRBTEST.COM (short username oryx)
I20251026 21:07:07.710305 11105 webserver.cc:492] Webserver started at http://127.3.175.60:45017/ using document root <none> and password file <none>
I20251026 21:07:07.710892 11105 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.710952 11105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.711154 11105 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:07.712970 11105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/instance:
uuid: "329609e082a84ad3bde9e588044d26c4"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "8221f8682e46008ba9440f6bb0319269"
server_key_iv: "d15dce40efd918fcb83cacb585db4c98"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.713474 11105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal/instance:
uuid: "329609e082a84ad3bde9e588044d26c4"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "8221f8682e46008ba9440f6bb0319269"
server_key_iv: "d15dce40efd918fcb83cacb585db4c98"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.717160 11105 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:07:07.719561 11198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.720655 11105 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:07:07.720793 11105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
uuid: "329609e082a84ad3bde9e588044d26c4"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "8221f8682e46008ba9440f6bb0319269"
server_key_iv: "d15dce40efd918fcb83cacb585db4c98"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.720904 11105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.733547 11105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:07.736917 11105 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:07.737145 11105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:07.744941 11105 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.60:36199
I20251026 21:07:07.745047 11250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.60:36199 every 8 connection(s)
I20251026 21:07:07.746049 11105 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/info.pb
I20251026 21:07:07.749207 11251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:07.753297 11251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } has no permanent_uuid. Determining permanent_uuid...
I20251026 21:07:07.753860 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 11105
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.60@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
I20251026 21:07:07.765672 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.755343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.3.175.62:41917)
1026 21:07:07.755475 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.755479 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.756040 (+ 561us) server_negotiation.cc:408] Connection header received
1026 21:07:07.756591 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.756595 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.756646 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.756741 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.758046 (+ 1305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.758534 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.759161 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.759390 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.761748 (+ 2358us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.761773 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.761780 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.761821 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.763637 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.764184 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.764187 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.764188 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.764232 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.764539 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.764542 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.764543 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.764698 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.764778 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.765433 (+ 655us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.765542 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 21:07:07.766453 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.755310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49920)
1026 21:07:07.755887 (+ 577us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.755929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.756406 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.757243 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.757247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.757611 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.757895 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.757905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.758644 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.758648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.759041 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.759060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.759279 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.759939 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.759961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.761567 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.763748 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.763755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.763764 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.764085 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.764312 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.764319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.764324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.764438 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.764831 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.764841 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.765138 (+ 297us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.765361 (+ 223us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.765651 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":68992,"thread_start_us":168,"threads_started":1}
I20251026 21:07:07.766618 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.755310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.3.175.60:36199)
1026 21:07:07.755777 (+ 467us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.755786 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.755827 (+ 41us) server_negotiation.cc:408] Connection header received
1026 21:07:07.756389 (+ 562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.756409 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.756615 (+ 206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.756783 (+ 168us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.757629 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.758384 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.759009 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.759805 (+ 796us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.762325 (+ 2520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.762352 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.762368 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.762403 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.764717 (+ 2314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.765148 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.765153 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.765155 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.765205 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.765431 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.765434 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.765435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.765639 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.765868 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.766310 (+ 442us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.766440 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":158,"threads_started":1}
I20251026 21:07:07.767732 11251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.60@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:07.775533 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.768245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58034)
1026 21:07:07.768365 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.768379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.768459 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.768881 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.768883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.769027 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.769206 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.769211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.770208 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.770210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.770583 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.770589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.770675 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.771205 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.771218 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.772525 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.774336 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.774339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.774341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.774496 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.774682 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.774684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.774685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.774722 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.775044 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.775047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.775128 (+ 81us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.775212 (+ 84us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.775374 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 21:07:07.775533 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.768358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.3.175.61:34739)
1026 21:07:07.768678 (+ 320us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.768681 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.768693 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:07.768729 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.768731 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.768773 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.768850 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.769342 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.770100 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.770715 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.770905 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.772646 (+ 1741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.772666 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.772669 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.772697 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.774221 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.774575 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.774578 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.774579 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.774619 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.774778 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.774780 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.774782 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.774962 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.775044 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.775258 (+ 214us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.775401 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20251026 21:07:07.776378 11251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.60@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
I20251026 21:07:07.783139 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.776876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40604)
1026 21:07:07.776986 (+ 110us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.776998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.777060 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.777274 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.777276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.777399 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.777549 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.777554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.778154 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.778156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.778473 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.778478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.778555 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.779086 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.779099 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.780288 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.781943 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.781946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.781947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.782162 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.782414 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.782417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.782418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.782463 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.782727 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.782729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.782798 (+ 69us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.782881 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.782987 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 21:07:07.783177 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.777003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.3.175.60:36199)
1026 21:07:07.777117 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.777120 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.777131 (+ 11us) server_negotiation.cc:408] Connection header received
1026 21:07:07.777158 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.777160 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.777195 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.777257 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.777627 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.778064 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.778549 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.778680 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.780413 (+ 1733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.780427 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.780429 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.780449 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.781829 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.782281 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.782284 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.782285 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.782324 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.782531 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.782533 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.782534 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.782666 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.782734 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.782915 (+ 181us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.783047 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":39}
I20251026 21:07:07.785899 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Bootstrap starting.
I20251026 21:07:07.788403 11251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:07.789172 11251 log.cc:826] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:07.791051 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: No bootstrap required, opened a new log
I20251026 21:07:07.793619 11251 raft_consensus.cc:359] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } }
I20251026 21:07:07.793815 11251 raft_consensus.cc:385] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:07.793884 11251 raft_consensus.cc:740] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329609e082a84ad3bde9e588044d26c4, State: Initialized, Role: FOLLOWER
I20251026 21:07:07.794364 11251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } }
I20251026 21:07:07.794840 11257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } } }
I20251026 21:07:07.794951 11257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:07.795573 11251 sys_catalog.cc:565] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 21:07:07.800189 11268 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 21:07:07.800280 11268 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 21:07:07.803731 11105 master_runner.cc:428] Detected that this master 329609e082a84ad3bde9e588044d26c4 is joining an existing cluster
I20251026 21:07:07.803805 11105 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.175.60:36199
I20251026 21:07:07.824374 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.821304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49938)
1026 21:07:07.821450 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.821470 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.821584 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.822025 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.822028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.822074 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:07.822252 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.822258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.823359 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.823364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.823988 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.823996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.824110 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.824162 (+ 52us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.824202 (+ 40us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.824257 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20251026 21:07:07.824911 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.821391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.3.175.62:41917)
1026 21:07:07.821757 (+ 366us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.821762 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.821776 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:07.821836 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.821839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.821890 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.821985 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:07.822361 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.823204 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.824119 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.824609 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.824678 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.824725 (+ 47us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.824787 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":171,"threads_started":1}
I20251026 21:07:07.829751 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.826235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58054)
1026 21:07:07.826358 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.826371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.826434 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.826983 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.826987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.827008 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:07.827207 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.827214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.828465 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.828469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.829364 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.829376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.829465 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.829489 (+ 24us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.829533 (+ 44us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.829603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251026 21:07:07.831395 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.826373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.3.175.61:34739)
1026 21:07:07.826669 (+ 296us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.826672 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.826684 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:07.826727 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.826729 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.826781 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.826977 (+ 196us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:07.827473 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.828306 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.830341 (+ 2035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.830794 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.830980 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.831072 (+ 92us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.831152 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20251026 21:07:07.835846 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40608)
1026 21:07:07.806701 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.806716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.806812 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.807312 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.807316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.827610 (+ 20294us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.827995 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.828008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.828959 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.828963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.829509 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.829518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.829627 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.830264 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.830280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.830766 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.834264 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.834268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.834270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.834659 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.834894 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.834897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.834899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.834945 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.835339 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.835356 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.835437 (+ 81us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.835536 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.835660 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":114,"threads_started":1}
I20251026 21:07:07.836300 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.3.175.60:36199)
1026 21:07:07.807006 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.807010 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.807022 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:07.807057 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.807060 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.807102 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.807185 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.828150 (+ 20965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.828806 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.829879 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.830082 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.830887 (+ 805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.830902 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.830905 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.830928 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.833179 (+ 2251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.834761 (+ 1582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.834764 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.834766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.834813 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.835022 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.835024 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.835027 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.835195 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.835983 (+ 788us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.836064 (+ 81us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.836166 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20251026 21:07:07.837383 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58044)
1026 21:07:07.807039 (+ 761us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.807053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.807185 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.807578 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.807584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.827665 (+ 20081us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.828303 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.828314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.829350 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.829354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.829958 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.829969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.830089 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.830700 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.830719 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.831152 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.834256 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.834260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.834262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.834683 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.835676 (+ 993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.835679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.835681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.835739 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.836775 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.836780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.836874 (+ 94us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.837034 (+ 160us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.837176 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":25,"thread_start_us":100,"threads_started":1}
I20251026 21:07:07.838296 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.3.175.61:34739)
1026 21:07:07.806476 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.806480 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.807113 (+ 633us) server_negotiation.cc:408] Connection header received
1026 21:07:07.807329 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.807334 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.807389 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.807439 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.828465 (+ 21026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.829197 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.830340 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.830538 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.831331 (+ 793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.831355 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.831361 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.831396 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.834123 (+ 2727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.835482 (+ 1359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.835488 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.835491 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.835550 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.835852 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.835859 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.835864 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.836621 (+ 757us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.836716 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.837924 (+ 1208us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.838099 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:07:07.840873 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.3.175.62:41917)
1026 21:07:07.806919 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.806924 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.807188 (+ 264us) server_negotiation.cc:408] Connection header received
1026 21:07:07.807388 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.807393 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.807447 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.807497 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.831849 (+ 24352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.832459 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.833599 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.833806 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.835008 (+ 1202us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.835029 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.835033 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.835063 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.839438 (+ 4375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.839808 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.839812 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.839814 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.839862 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.840063 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.840066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.840068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.840248 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.840347 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.840598 (+ 251us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.840716 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20251026 21:07:07.841210 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.806792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49928)
1026 21:07:07.807116 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.807130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.807246 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.807632 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.807637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.831374 (+ 23737us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.831673 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.831685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.832962 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.832967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.833474 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.833483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.833613 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.834216 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.834240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.834864 (+ 624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.839539 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.839542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.839543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.839717 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.839942 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.839944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.839946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.839986 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.840327 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.840329 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.840403 (+ 74us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.840519 (+ 116us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.840655 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.62@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
I20251026 21:07:07.853751 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.845407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40610)
1026 21:07:07.845618 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.845631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.845713 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.846020 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.846022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.846215 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.846409 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.846415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.847375 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.847378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.847829 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.847841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.847956 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.848672 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.848690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.850361 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.852525 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.852528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.852530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.852721 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.852947 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.852949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.852951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.852991 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.853339 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.853341 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.853421 (+ 80us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.853497 (+ 76us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.853617 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20251026 21:07:07.854163 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.845520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.3.175.60:36199)
1026 21:07:07.845651 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.845656 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.845671 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:07.845854 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.845858 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.845917 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.846007 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.846515 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.847218 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.847948 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.848174 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.850465 (+ 2291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.850486 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.850489 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.850521 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.852391 (+ 1870us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.852815 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.852819 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.852821 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.852870 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.853066 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.853070 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.853072 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.853259 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.853358 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.853557 (+ 199us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.853689 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:07:07.855191 10521 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.3.175.60:36199: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "329609e082a84ad3bde9e588044d26c4"
member_type: NON_VOTER
last_known_addr {
host: "127.3.175.60"
port: 36199
}
attrs {
promote: true
}
}
dest_uuid: "1c2e62502e21436a81044f938290c8f2"
cas_config_opid_index: 5
I20251026 21:07:07.855798 10521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } }
I20251026 21:07:07.857074 11092 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 329609e082a84ad3bde9e588044d26c4 (127.3.175.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.857399 11225 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:07.858232 11092 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.3.175.60:36199
I20251026 21:07:07.858428 11092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.858515 11092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:07.858809 11225 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 21:07:07.859016 11105 master.cc:561] Master@127.3.175.60:36199 shutting down...
I20251026 21:07:07.859381 11135 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 LEARNER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 21:07:07.859895 11091 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:07.860838 11091 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 21:07:07.863020 11135 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 329609e082a84ad3bde9e588044d26c4 (127.3.175.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
W20251026 21:07:07.865047 10506 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 -> Peer 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199): Couldn't send request to peer 329609e082a84ad3bde9e588044d26c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 21:07:07.865479 11280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.865720 11280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: LEARNER
I20251026 21:07:07.865937 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c2e62502e21436a81044f938290c8f2. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: NON_VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: true } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.866053 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: LEARNER
I20251026 21:07:07.866811 11091 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: attempting to promote NON_VOTER 02b06b2ce5944ee9b05df06891a13b35 to VOTER
I20251026 21:07:07.867126 11105 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251026 21:07:07.867424 10521 master.cc:679] Remote error: unable to get registration information for peer 329609e082a84ad3bde9e588044d26c4: Service unavailable: service kudu.master.MasterService not registered on Master
I20251026 21:07:07.867372 11091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } }
I20251026 21:07:07.867448 11105 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:07.867547 11105 tablet_replica.cc:333] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: stopping tablet replica
I20251026 21:07:07.868669 11135 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 LEARNER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251026 21:07:07.869067 11092 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251026 21:07:07.870707 10506 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 -> Peer 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199): Couldn't send request to peer 329609e082a84ad3bde9e588044d26c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 21:07:07.870813 11092 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.872258 11092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.872365 11092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:07.872478 11092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.872545 11092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:07.873015 11135 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.875319 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:07.875440 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:07.886022 11105 master.cc:583] Master@127.3.175.60:36199 shutdown complete.
I20251026 21:07:07.886107 11105 master_runner.cc:305] Clearing existing system tablet
I20251026 21:07:07.886355 11105 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.886425 11105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.888324 11105 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.889297 11283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.889468 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.889144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40620)
1026 21:07:07.889291 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.889374 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.60:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 21:07:07.889490 11105 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.889628 11105 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.889670 11105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
uuid: "329609e082a84ad3bde9e588044d26c4"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "8221f8682e46008ba9440f6bb0319269"
server_key_iv: "d15dce40efd918fcb83cacb585db4c98"
server_key_version: "encryptionkey@0"
W20251026 21:07:07.889869 10521 master.cc:679] Network error: unable to get registration information for peer 329609e082a84ad3bde9e588044d26c4: Client connection negotiation failed: client connection to 127.3.175.60:36199: connect: Connection refused (error 111)
I20251026 21:07:07.891718 11105 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:07.893544 11105 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251026 21:07:07.893635 11105 log.cc:1199] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251026 21:07:07.893905 11105 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Deleting consensus metadata
I20251026 21:07:07.894079 11105 master_runner.cc:315] Copying system tablet from 127.3.175.62:41917
I20251026 21:07:07.895112 11105 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: tablet copy: Beginning tablet copy session from remote peer at address 127.3.175.62:41917
I20251026 21:07:07.902552 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.895825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.3.175.62:41917)
1026 21:07:07.895981 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.895986 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.896083 (+ 97us) server_negotiation.cc:408] Connection header received
1026 21:07:07.896243 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.896246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.896299 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.896366 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.897052 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.897657 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.898313 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.898509 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.899544 (+ 1035us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:07.899563 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:07.899566 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:07.899591 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:07.901079 (+ 1488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.901452 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.901456 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.901457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.901501 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:07.901715 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:07.901718 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:07.901720 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:07.901881 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:07.901969 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:07.902277 (+ 308us) server_negotiation.cc:300] Negotiation successful
1026 21:07:07.902400 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251026 21:07:07.902570 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.895703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49954)
1026 21:07:07.895981 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.896000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.896135 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.896400 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.896403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.896683 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.896896 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.896903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.897771 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.897775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.898200 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.898208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.898323 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:07.899000 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:07.899014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:07.899357 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:07.901192 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.901195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.901198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.901360 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.901588 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:07.901591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:07.901593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:07.901636 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:07.901969 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:07.901973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:07.902112 (+ 139us) client_negotiation.cc:769] Sending connection context
1026 21:07:07.902208 (+ 96us) client_negotiation.cc:240] Negotiation successful
1026 21:07:07.902369 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20251026 21:07:07.903237 10541 tablet_copy_service.cc:140] P 1c2e62502e21436a81044f938290c8f2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 329609e082a84ad3bde9e588044d26c4 ({username='oryx', principal='oryx/127.3.175.60@KRBTEST.COM'} at 127.0.0.1:49954)
I20251026 21:07:07.903378 10541 tablet_copy_service.cc:161] P 1c2e62502e21436a81044f938290c8f2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 329609e082a84ad3bde9e588044d26c4 at {username='oryx', principal='oryx/127.3.175.60@KRBTEST.COM'} at 127.0.0.1:49954: session id = 329609e082a84ad3bde9e588044d26c4-00000000000000000000000000000000
I20251026 21:07:07.904470 10541 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 21:07:07.905864 11105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251026 21:07:07.907822 11105 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: tablet copy: Starting download of 0 data blocks...
I20251026 21:07:07.908008 11105 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: tablet copy: Starting download of 1 WAL segments...
I20251026 21:07:07.910075 11105 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 21:07:07.912185 10541 tablet_copy_service.cc:342] P 1c2e62502e21436a81044f938290c8f2: Request end of tablet copy session 329609e082a84ad3bde9e588044d26c4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.175.60@KRBTEST.COM'} at 127.0.0.1:49954
I20251026 21:07:07.912287 10541 tablet_copy_service.cc:434] P 1c2e62502e21436a81044f938290c8f2: ending tablet copy session 329609e082a84ad3bde9e588044d26c4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 329609e082a84ad3bde9e588044d26c4
W20251026 21:07:07.917690 11105 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251026 21:07:07.920601 11105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:07.922562 11290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.925894 11293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:07.925899 11291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:07.926486 11105 server_base.cc:1047] running on GCE node
I20251026 21:07:07.926654 11105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:07.926950 11105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:07.928068 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.927762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40634)
1026 21:07:07.927909 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.927970 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.60:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 21:07:07.928088 11105 hybrid_clock.cc:648] HybridClock initialized: now 1761512827928079 us; error 25 us; skew 500 ppm
W20251026 21:07:07.928287 10521 master.cc:679] Network error: unable to get registration information for peer 329609e082a84ad3bde9e588044d26c4: Client connection negotiation failed: client connection to 127.3.175.60:36199: connect: Connection refused (error 111)
Oct 26 21:07:07 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:07.929716 11105 init.cc:377] Logged in from keytab as oryx/127.3.175.60@KRBTEST.COM (short username oryx)
I20251026 21:07:07.930205 11105 webserver.cc:492] Webserver started at http://127.3.175.60:41247/ using document root <none> and password file <none>
I20251026 21:07:07.930382 11105 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:07.930421 11105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:07.931669 11105 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.932597 11300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:07.932924 11105 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:07.932997 11105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
uuid: "329609e082a84ad3bde9e588044d26c4"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "8221f8682e46008ba9440f6bb0319269"
server_key_iv: "d15dce40efd918fcb83cacb585db4c98"
server_key_version: "encryptionkey@0"
I20251026 21:07:07.933228 11105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:07.942999 10850 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
I20251026 21:07:07.980928 11105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:07.983198 11105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:07.988273 11105 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.60:36199
I20251026 21:07:07.988554 11352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.60:36199 every 8 connection(s)
I20251026 21:07:07.988976 11105 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/master-2/data/info.pb
I20251026 21:07:07.990419 11353 sys_catalog.cc:263] Verifying existing consensus state
I20251026 21:07:07.991375 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Bootstrap starting.
I20251026 21:07:08.002416 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 21:07:08.002815 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: Bootstrap complete.
I20251026 21:07:08.003621 11353 raft_consensus.cc:359] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } }
I20251026 21:07:08.003777 11353 raft_consensus.cc:740] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 329609e082a84ad3bde9e588044d26c4, State: Initialized, Role: LEARNER
I20251026 21:07:08.003902 11353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } }
I20251026 21:07:08.004259 11357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:08.004433 11353 sys_catalog.cc:565] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:08.004432 11357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: LEARNER
I20251026 21:07:08.005389 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.996651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40638)
1026 21:07:07.996832 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:07.996849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:07.996958 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:07.997436 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:07.997440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:07.997645 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:07.997914 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.997925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.999175 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.999178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:07.999806 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:07.999816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.999933 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:08.000761 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:08.000778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:08.001184 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:08.003671 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:08.003678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:08.003683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:08.003931 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:08.004212 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:08.004215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:08.004217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:08.004280 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:08.004711 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:08.004714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:08.004829 (+ 115us) client_negotiation.cc:769] Sending connection context
1026 21:07:08.004968 (+ 139us) client_negotiation.cc:240] Negotiation successful
1026 21:07:08.005156 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20251026 21:07:08.005414 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:07.996736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.3.175.60:36199)
1026 21:07:07.997181 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:07.997185 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:07.997199 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:07.997241 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:07.997244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:07.997299 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:07.997397 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:07.998065 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:07.999048 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:07.999933 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:08.000171 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:08.001320 (+ 1149us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:08.001358 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:08.001361 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:08.001387 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:08.003525 (+ 2138us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:08.004053 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:08.004056 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:08.004058 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:08.004108 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:08.004385 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:08.004387 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:08.004389 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:08.004613 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:08.004713 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:08.005092 (+ 379us) server_negotiation.cc:300] Negotiation successful
1026 21:07:08.005226 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":116,"threads_started":1}
W20251026 21:07:08.006474 10521 master.cc:679] Service unavailable: unable to get registration information for peer 329609e082a84ad3bde9e588044d26c4: Catalog manager is not initialized. State: Starting
I20251026 21:07:08.007666 11368 catalog_manager.cc:1269] Loaded cluster ID: a744a34c9b134fdc866f41e652cc9d9b
I20251026 21:07:08.007728 11368 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: loading cluster ID for follower catalog manager: success
I20251026 21:07:08.009438 11368 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: acquiring CA information for follower catalog manager: success
I20251026 21:07:08.010651 11368 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 21:07:08.177428 10986 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
I20251026 21:07:08.463357 11327 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251026 21:07:08.463804 11357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c2e62502e21436a81044f938290c8f2. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:08.463932 11357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: LEARNER
I20251026 21:07:08.991464 11281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: NON_VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: true } } }
I20251026 21:07:08.991648 11281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:08.992524 11281 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: attempting to promote NON_VOTER 329609e082a84ad3bde9e588044d26c4 to VOTER
I20251026 21:07:08.992885 11281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:08.994094 11327 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 LEARNER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251026 21:07:08.994134 11135 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Refusing update from remote peer 1c2e62502e21436a81044f938290c8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251026 21:07:08.994508 11281 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251026 21:07:08.994728 11370 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251026 21:07:08.997007 11281 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 329609e082a84ad3bde9e588044d26c4 (127.3.175.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.997610 11135 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 329609e082a84ad3bde9e588044d26c4 (127.3.175.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.998682 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.998711 11281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.998797 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:08.998821 11281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:08.998940 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.998652 11327 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 329609e082a84ad3bde9e588044d26c4 (127.3.175.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.999027 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:08.999433 11375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:08.999552 11375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:09.000490 11376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c2e62502e21436a81044f938290c8f2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:09.000614 11376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:09.932262 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.921786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.3.175.61:34739)
1026 21:07:09.922291 (+ 505us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.922295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.922305 (+ 10us) server_negotiation.cc:408] Connection header received
1026 21:07:09.922337 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.922339 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.922378 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.922447 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:09.923480 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.924013 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.924604 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.924727 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.929250 (+ 4523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:09.929267 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:09.929270 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:09.929302 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:09.930843 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:09.931209 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:09.931211 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:09.931213 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:09.931251 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:09.931449 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:09.931451 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:09.931453 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:09.931604 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:09.931704 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.931916 (+ 212us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.932075 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":91,"threads_started":1}
I20251026 21:07:09.935727 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.933134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49962)
1026 21:07:09.933372 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.933383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.933454 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.933701 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.933704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.933720 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.933970 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.933981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.934774 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.934776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.935412 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.935420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.935511 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.935529 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.935564 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.935616 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20251026 21:07:09.936204 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.933272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.3.175.62:41917)
1026 21:07:09.933505 (+ 233us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.933509 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.933520 (+ 11us) server_negotiation.cc:408] Connection header received
1026 21:07:09.933562 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.933564 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.933606 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.933679 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.934129 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.934669 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.935531 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.935935 (+ 404us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.935982 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.936028 (+ 46us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.936089 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20251026 21:07:09.939404 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.937370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58076)
1026 21:07:09.937492 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.937504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.937565 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.937754 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.937756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.937767 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.937926 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.937931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.938539 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.938541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.939082 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.939088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.939179 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.939193 (+ 14us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.939225 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.939276 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251026 21:07:09.939977 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.937429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.3.175.61:34739)
1026 21:07:09.937530 (+ 101us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.937533 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.937545 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:09.937646 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.937648 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.937682 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.937742 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.938006 (+ 264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.938463 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.939265 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.939770 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.939802 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.939849 (+ 47us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.939896 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 21:07:09.943449 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.941083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40648)
1026 21:07:09.941205 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.941217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.941276 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.941655 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.941657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.941669 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.941804 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.941808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.942580 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.942583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.943139 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.943146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.943224 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.943236 (+ 12us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.943273 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.943358 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 21:07:09.944039 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.941199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.3.175.60:36199)
1026 21:07:09.941437 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.941440 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.941453 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:09.941497 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.941499 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.941551 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.941645 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.941938 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.942475 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.943251 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.943705 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.943824 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.943885 (+ 61us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.943942 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20251026 21:07:09.952677 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.945787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.3.175.60:36199)
1026 21:07:09.945934 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.945939 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.945951 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:09.946049 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.946053 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.946092 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.946158 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:09.946926 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.947574 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.948296 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.948429 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.949556 (+ 1127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:09.949572 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:09.949574 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:09.949592 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:09.951204 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:09.951630 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:09.951634 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:09.951635 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:09.951690 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:09.951942 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:09.951946 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:09.951947 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:09.952102 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:09.952216 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.952380 (+ 164us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.952487 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 21:07:09.956154 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.953524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:49974)
1026 21:07:09.953789 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.953805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.953913 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.954156 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.954159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.954177 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.954363 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.954368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.955129 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.955132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.955735 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.955743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.955853 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.955875 (+ 22us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.955928 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.955994 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20251026 21:07:09.956583 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.953657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.3.175.62:41917)
1026 21:07:09.953798 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.953805 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.953873 (+ 68us) server_negotiation.cc:408] Connection header received
1026 21:07:09.954006 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.954009 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.954061 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.954130 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.954471 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.955011 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.955859 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.956335 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.956375 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.956432 (+ 57us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.956494 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:07:09.960111 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.957915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.0.0.1:58082)
1026 21:07:09.958048 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.958060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.958142 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.958384 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.958386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.958398 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.958550 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.958554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.959221 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.959223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.959820 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.959828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.959914 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.959931 (+ 17us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.959967 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.960009 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251026 21:07:09.960579 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.958076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.3.175.61:34739)
1026 21:07:09.958198 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.958202 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.958215 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:09.958258 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.958260 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.958303 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.958374 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.958628 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.959138 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.959937 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.960335 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.960384 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.960440 (+ 56us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.960488 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:07:09.963994 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.961808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.0.0.1:40656)
1026 21:07:09.961934 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:09.961950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:09.962026 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:09.962194 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:09.962196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:09.962208 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:09.962365 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.962374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.963103 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.963106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:09.963695 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:09.963702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.963794 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.963812 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:07:09.963851 (+ 39us) client_negotiation.cc:240] Negotiation successful
1026 21:07:09.963899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251026 21:07:09.964442 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:09.961918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.3.175.60:36199)
1026 21:07:09.962026 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:09.962030 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:09.962043 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:09.962082 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:09.962084 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:09.962121 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:09.962184 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:09.962460 (+ 276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.962993 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:09.963834 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:09.964233 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:09.964276 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:09.964314 (+ 38us) server_negotiation.cc:300] Negotiation successful
1026 21:07:09.964358 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":1792}
I20251026 21:07:09.965907 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10585
I20251026 21:07:09.972308 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:40987
--local_ip_for_outbound_sockets=127.3.175.1
--tserver_master_addrs=127.3.175.62:41917
--webserver_port=46579
--webserver_interface=127.3.175.1
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:10.087177 11383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:10.087497 11383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:10.087587 11383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:10.090984 11383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:10.091059 11383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:10.091171 11383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:10.095628 11383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:40987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=46579
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:10.096817 11383 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:10.097731 11383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:10.104614 11389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:10.104622 11388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:10.104719 11383 server_base.cc:1047] running on GCE node
W20251026 21:07:10.104633 11391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:10.105352 11383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:10.105950 11383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:10.107105 11383 hybrid_clock.cc:648] HybridClock initialized: now 1761512830107090 us; error 30 us; skew 500 ppm
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:10.110066 11383 init.cc:377] Logged in from keytab as oryx/127.3.175.1@KRBTEST.COM (short username oryx)
I20251026 21:07:10.111212 11383 webserver.cc:492] Webserver started at http://127.3.175.1:46579/ using document root <none> and password file <none>
I20251026 21:07:10.111799 11383 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:10.111867 11383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:10.115644 11383 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20251026 21:07:10.117533 11398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.118582 11383 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:10.118682 11383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "643943dcd2704d72852a37536e03a82f"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "972ffb961053daa646fcf7ff42e0baa2"
server_key_iv: "8710288f09ec97715cfeec0457a957e3"
server_key_version: "encryptionkey@0"
I20251026 21:07:10.119097 11383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:10.154719 11383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:10.157951 11383 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:10.158157 11383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:10.158738 11383 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:10.159760 11383 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:10.159812 11383 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.159869 11383 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:10.159900 11383 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.169006 11383 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:40987
I20251026 21:07:10.169034 11511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:40987 every 8 connection(s)
I20251026 21:07:10.170092 11383 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:10.178669 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 11383
I20251026 21:07:10.178906 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10721
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
I20251026 21:07:10.185475 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.171953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:47975 (local address 127.3.175.60:36199)
1026 21:07:10.172103 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.172107 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.173037 (+ 930us) server_negotiation.cc:408] Connection header received
1026 21:07:10.174195 (+ 1158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.174199 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.174263 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.174316 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.176084 (+ 1768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.176594 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.177367 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177557 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.181091 (+ 3534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.181128 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.181131 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.181171 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.183901 (+ 2730us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.184295 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.184301 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.184303 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.184348 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.184557 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.184560 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.184562 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.184741 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.184842 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.185161 (+ 319us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.185294 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:07:10.185613 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.172460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:55255 (local address 127.3.175.62:41917)
1026 21:07:10.172612 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.172617 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.173412 (+ 795us) server_negotiation.cc:408] Connection header received
1026 21:07:10.174172 (+ 760us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.174175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.174241 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.174323 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.176081 (+ 1758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.176890 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.177614 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177762 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.181094 (+ 3332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.181115 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.181119 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.181148 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.183199 (+ 2051us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.183751 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.183756 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.183757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.183804 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.184113 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.184120 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.184125 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.184353 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.184448 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.185210 (+ 762us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.185373 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:07:10.186587 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.172295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.3.175.1:47975)
1026 21:07:10.172893 (+ 598us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.172936 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.173979 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.174498 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.174505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.175600 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.175914 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.175923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.176736 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.176738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.177245 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.177253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177480 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.178231 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.178254 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.180906 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.184009 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.184013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.184014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.184200 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.184432 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.184435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.184437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.184480 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.184831 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.184834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.184945 (+ 111us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.185128 (+ 183us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.185430 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":120,"threads_started":1}
I20251026 21:07:10.186606 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.172351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:55255)
1026 21:07:10.173309 (+ 958us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.173322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.173977 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.174496 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.174504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.175096 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.175906 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.175921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177035 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.177038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.177477 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.177485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177907 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.178427 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.178446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.180886 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.183347 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.183354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.183368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.183650 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.183896 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.183898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.183900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.184009 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.184514 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.184524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.184879 (+ 355us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.185830 (+ 951us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.185999 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":180,"spinlock_wait_cycles":5888,"thread_start_us":73,"threads_started":1}
I20251026 21:07:10.187927 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:41823
--local_ip_for_outbound_sockets=127.3.175.2
--tserver_master_addrs=127.3.175.62:41917
--webserver_port=37501
--webserver_interface=127.3.175.2
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 21:07:10.188100 11512 heartbeater.cc:344] Connected to a master server at 127.3.175.60:36199
I20251026 21:07:10.188367 11512 heartbeater.cc:461] Registering TS with master...
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:10.189039 11512 heartbeater.cc:507] Master 127.3.175.60:36199 requested a full tablet report, sending...
I20251026 21:07:10.189163 11513 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:10.189260 11513 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.189484 11513 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:10.190639 10521 ts_manager.cc:194] Re-registered known tserver with Master: 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
I20251026 21:07:10.190764 11317 ts_manager.cc:194] Registered new tserver with Master: 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
I20251026 21:07:10.191445 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:55255
I20251026 21:07:10.194057 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.172295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.3.175.1:36521)
1026 21:07:10.173177 (+ 882us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.173189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.173966 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.174527 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.174529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.175144 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.175977 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.175987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177000 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.177006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.177514 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.177521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177975 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.187399 (+ 9424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.187418 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.188889 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.191543 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.191548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.191550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.191756 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.193115 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.193118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.193120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.193164 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.193631 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.193633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.193699 (+ 66us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.193777 (+ 78us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.193917 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":27,"spinlock_wait_cycles":47744,"thread_start_us":136,"threads_started":1}
I20251026 21:07:10.194238 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.171952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:36521 (local address 127.3.175.61:34739)
1026 21:07:10.172103 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.172114 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.173271 (+ 1157us) server_negotiation.cc:408] Connection header received
1026 21:07:10.174297 (+ 1026us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.174303 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.174359 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.174431 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.176093 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.176879 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.177652 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.177868 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.189078 (+ 11210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.189103 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.189106 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.189142 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.191336 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.191892 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.191896 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.191898 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.191953 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.193270 (+ 1317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.193276 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.193279 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.193515 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.193657 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.193859 (+ 202us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.194006 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:07:10.195044 11514 heartbeater.cc:344] Connected to a master server at 127.3.175.61:34739
I20251026 21:07:10.195161 11514 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.195443 11514 heartbeater.cc:507] Master 127.3.175.61:34739 requested a full tablet report, sending...
I20251026 21:07:10.196650 11133 ts_manager.cc:194] Registered new tserver with Master: 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
W20251026 21:07:10.302529 11519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:10.302798 11519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:10.302877 11519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:10.306607 11519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:10.306677 11519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:10.306806 11519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:10.311666 11519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:41823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=37501
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:10.312812 11519 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:10.313671 11519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:10.320497 11527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:10.320497 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:10.320784 11519 server_base.cc:1047] running on GCE node
W20251026 21:07:10.320500 11525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:10.321249 11519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:10.321807 11519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:10.322970 11519 hybrid_clock.cc:648] HybridClock initialized: now 1761512830322958 us; error 27 us; skew 500 ppm
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:10.326018 11519 init.cc:377] Logged in from keytab as oryx/127.3.175.2@KRBTEST.COM (short username oryx)
I20251026 21:07:10.327133 11519 webserver.cc:492] Webserver started at http://127.3.175.2:37501/ using document root <none> and password file <none>
I20251026 21:07:10.327739 11519 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:10.327809 11519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:10.331594 11519 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20251026 21:07:10.333544 11534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.334758 11519 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:10.334888 11519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "226478220f964d02baead480f14a9c72"
format_stamp: "Formatted at 2025-10-26 21:07:06 on dist-test-slave-th6x"
server_key: "75b06b237124bd69f2fe90af37a2c64b"
server_key_iv: "00c30211a720e57bb80501da8589a53f"
server_key_version: "encryptionkey@0"
I20251026 21:07:10.335440 11519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:10.365610 11519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:10.368799 11519 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:10.368970 11519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:10.369494 11519 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:10.370405 11519 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:10.370453 11519 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.370496 11519 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:10.370517 11519 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.379571 11519 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:41823
I20251026 21:07:10.379603 11647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:41823 every 8 connection(s)
I20251026 21:07:10.380581 11519 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:10.385315 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 11519
I20251026 21:07:10.385502 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10857
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:10.394766 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:42287
--local_ip_for_outbound_sockets=127.3.175.3
--tserver_master_addrs=127.3.175.62:41917
--webserver_port=33487
--webserver_interface=127.3.175.3
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199 with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 21:07:10.398229 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:41675 (local address 127.3.175.61:34739)
1026 21:07:10.382717 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.382724 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.383522 (+ 798us) server_negotiation.cc:408] Connection header received
1026 21:07:10.384817 (+ 1295us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.384821 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.384911 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.384966 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.387519 (+ 2553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.388266 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.389121 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.389312 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.393298 (+ 3986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.393332 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.393336 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.393370 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.395934 (+ 2564us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396507 (+ 573us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.396513 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.396516 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.396574 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396829 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.396833 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.396835 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.397363 (+ 528us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.397458 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.397878 (+ 420us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.398019 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 21:07:10.398252 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:58719 (local address 127.3.175.60:36199)
1026 21:07:10.383095 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.383102 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.383970 (+ 868us) server_negotiation.cc:408] Connection header received
1026 21:07:10.384811 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.384816 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.384890 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.384950 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.387187 (+ 2237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.388202 (+ 1015us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.389244 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.389479 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.393341 (+ 3862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.393361 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.393364 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.393393 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.395707 (+ 2314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396379 (+ 672us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.396384 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.396386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.396439 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396783 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.396786 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.396788 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.397010 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.397100 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.397890 (+ 790us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.398028 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:10.399860 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.3.175.2:58719)
1026 21:07:10.383856 (+ 960us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.383878 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.384568 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.385477 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.385481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.386138 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.386988 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.387008 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.388371 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.388378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.389097 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.389110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.389616 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.390337 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.390376 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.393119 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.395850 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.395856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.395882 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.396256 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.396547 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.396552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.396686 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.397193 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.397202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.397547 (+ 345us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.398526 (+ 979us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.398819 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":88,"spinlock_wait_cycles":32000,"thread_start_us":152,"threads_started":1}
I20251026 21:07:10.400000 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.3.175.2:41675)
1026 21:07:10.383382 (+ 562us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.383425 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.384578 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.385378 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.385389 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.386165 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.387370 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.387380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.388405 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.388408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.388987 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.388996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.389575 (+ 579us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.390398 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.390415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.393100 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.396068 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.396079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.396370 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.396674 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.396676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.396677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.396729 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.397516 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.397520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.397599 (+ 79us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.398453 (+ 854us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.398918 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":137,"spinlock_wait_cycles":6016,"thread_start_us":124,"threads_started":1}
I20251026 21:07:10.401439 11650 heartbeater.cc:344] Connected to a master server at 127.3.175.61:34739
I20251026 21:07:10.401448 11648 heartbeater.cc:344] Connected to a master server at 127.3.175.60:36199
I20251026 21:07:10.401713 11648 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.401714 11650 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.402406 11648 heartbeater.cc:507] Master 127.3.175.60:36199 requested a full tablet report, sending...
I20251026 21:07:10.403316 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:57785)
1026 21:07:10.383665 (+ 845us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.383678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.384568 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.394329 (+ 9761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.394334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.394524 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.394784 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.394791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.395971 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.395974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.396578 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.396584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.396739 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.397297 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.397313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.399180 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.401835 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.401839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.401842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.402055 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.402348 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.402351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.402353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.402404 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.402790 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.402795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.402885 (+ 90us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.402976 (+ 91us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.403110 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":30,"spinlock_wait_cycles":6272,"thread_start_us":85,"threads_started":1}
I20251026 21:07:10.403321 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.382438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:57785 (local address 127.3.175.62:41917)
1026 21:07:10.382598 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.382603 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.383744 (+ 1141us) server_negotiation.cc:408] Connection header received
1026 21:07:10.384811 (+ 1067us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.384815 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.384899 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.384952 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.394966 (+ 10014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.395836 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.396729 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.396959 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.399364 (+ 2405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.399388 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.399391 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.399425 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.401623 (+ 2198us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.402176 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.402180 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.402182 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.402235 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.402489 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.402493 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.402495 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.402678 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.402787 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.403037 (+ 250us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.403162 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251026 21:07:10.403853 11317 ts_manager.cc:194] Registered new tserver with Master: 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:10.404284 11649 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:10.404367 11649 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.402407 11650 heartbeater.cc:507] Master 127.3.175.61:34739 requested a full tablet report, sending...
I20251026 21:07:10.404572 11649 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:10.405129 10521 ts_manager.cc:194] Re-registered known tserver with Master: 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:10.405180 11133 ts_manager.cc:194] Registered new tserver with Master: 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:10.405776 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:57785
W20251026 21:07:10.508504 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:10.508755 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:10.508816 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:10.512349 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:10.512427 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:10.512511 11655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:10.516913 11655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:42287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=33487
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:41917,127.3.175.61:34739,127.3.175.60:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:10.518040 11655 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:10.518932 11655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:10.526093 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:10.526098 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:10.526093 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:10.526332 11655 server_base.cc:1047] running on GCE node
I20251026 21:07:10.526743 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:10.527397 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:10.528550 11655 hybrid_clock.cc:648] HybridClock initialized: now 1761512830528539 us; error 32 us; skew 500 ppm
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:10.531524 11655 init.cc:377] Logged in from keytab as oryx/127.3.175.3@KRBTEST.COM (short username oryx)
I20251026 21:07:10.532668 11655 webserver.cc:492] Webserver started at http://127.3.175.3:33487/ using document root <none> and password file <none>
I20251026 21:07:10.533257 11655 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:10.533321 11655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:10.537083 11655 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20251026 21:07:10.538980 11670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.540132 11655 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:10.540230 11655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "8b0950373eb54a7886e1314b21421055"
format_stamp: "Formatted at 2025-10-26 21:07:07 on dist-test-slave-th6x"
server_key: "bb4f3d30df36b76a0fc1d3caae527c22"
server_key_iv: "11a364b082cef1836dbab89c0256bddb"
server_key_version: "encryptionkey@0"
I20251026 21:07:10.540609 11655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:10.574252 11655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:10.577459 11655 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:10.577623 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:10.578137 11655 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:10.579025 11655 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:10.579072 11655 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.579113 11655 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:10.579131 11655 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:10.588083 11655 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:42287
I20251026 21:07:10.588099 11783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:42287 every 8 connection(s)
I20251026 21:07:10.589179 11655 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:10.592804 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 11655
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.60@KRBTEST.COM
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
Oct 26 21:07:10 dist-test-slave-th6x krb5kdc[10473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512830, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.61@KRBTEST.COM
I20251026 21:07:10.605585 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:54727 (local address 127.3.175.60:36199)
1026 21:07:10.591650 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.591655 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.592343 (+ 688us) server_negotiation.cc:408] Connection header received
1026 21:07:10.593013 (+ 670us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.593020 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.593088 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.593167 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.595045 (+ 1878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.595829 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.597310 (+ 1481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.597527 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.600038 (+ 2511us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.600057 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.600061 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.600088 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.602303 (+ 2215us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.602916 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.602921 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.602923 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.602974 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.604585 (+ 1611us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.604588 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.604590 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.604764 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.604882 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.605286 (+ 404us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.605405 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:07:10.605630 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:37283 (local address 127.3.175.61:34739)
1026 21:07:10.591189 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.591194 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.592312 (+ 1118us) server_negotiation.cc:408] Connection header received
1026 21:07:10.593078 (+ 766us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.593082 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.593154 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.593209 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.596488 (+ 3279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.597810 (+ 1322us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.598662 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.598845 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.601370 (+ 2525us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.601392 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.601395 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.601427 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.603387 (+ 1960us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.603818 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.603822 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.603825 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.603892 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.604213 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.604217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.604220 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.604415 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.604508 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.605327 (+ 819us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.605455 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:07:10.606551 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.61:34739 (local address 127.3.175.3:37283)
1026 21:07:10.592206 (+ 993us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.592219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.592801 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.593358 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.593367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.594079 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.596339 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.596348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.597966 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.597970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.598528 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.598539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.598981 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.599539 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.599558 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.601243 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.603479 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.603482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.603484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.603701 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.604019 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.604024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.604027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.604095 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.604582 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.604591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.604998 (+ 407us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.605928 (+ 930us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.606072 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":71,"spinlock_wait_cycles":54272,"thread_start_us":146,"threads_started":1}
I20251026 21:07:10.606559 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.60:36199 (local address 127.3.175.3:54727)
1026 21:07:10.592253 (+ 833us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.592266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.592805 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.593358 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.593368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.594052 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.594879 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.594900 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.595959 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.595966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.596573 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.596581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.596778 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.597498 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.597530 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.599493 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.602423 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.602429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.602443 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.602815 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.603079 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.603083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.603084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.603220 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.604882 (+ 1662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.604887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.605015 (+ 128us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.605250 (+ 235us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.605543 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"spinlock_wait_cycles":45952,"thread_start_us":103,"threads_started":1}
I20251026 21:07:10.608253 11784 heartbeater.cc:344] Connected to a master server at 127.3.175.60:36199
I20251026 21:07:10.608376 11786 heartbeater.cc:344] Connected to a master server at 127.3.175.61:34739
I20251026 21:07:10.608536 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:58711)
1026 21:07:10.591889 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.591930 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.592790 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.593882 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.593886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.594273 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:10.595009 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.595020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.596031 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.596035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.596622 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.596630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.597046 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.597703 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:10.597719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:10.603130 (+ 5411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:10.606247 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.606251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.606253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.606448 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.607343 (+ 895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:10.607348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:10.607352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:10.607414 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:10.608010 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:10.608013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:10.608107 (+ 94us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.608206 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.608342 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":129,"threads_started":1}
I20251026 21:07:10.608538 11784 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.608922 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.591000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:58711 (local address 127.3.175.62:41917)
1026 21:07:10.591162 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.591168 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.592094 (+ 926us) server_negotiation.cc:408] Connection header received
1026 21:07:10.593593 (+ 1499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.593597 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.593656 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.593737 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.595169 (+ 1432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.595892 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.596741 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.596928 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.603273 (+ 6345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.603321 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.603325 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.603358 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.605916 (+ 2558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.607129 (+ 1213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.607133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.607135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.607197 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.607507 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.607513 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.607516 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.607719 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.607817 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.608671 (+ 854us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.608798 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251026 21:07:10.609248 11784 heartbeater.cc:507] Master 127.3.175.60:36199 requested a full tablet report, sending...
I20251026 21:07:10.609613 11785 heartbeater.cc:344] Connected to a master server at 127.3.175.62:41917
I20251026 21:07:10.609714 11785 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.608538 11786 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:10.609946 11785 heartbeater.cc:507] Master 127.3.175.62:41917 requested a full tablet report, sending...
I20251026 21:07:10.609951 11786 heartbeater.cc:507] Master 127.3.175.61:34739 requested a full tablet report, sending...
I20251026 21:07:10.610726 10521 ts_manager.cc:194] Re-registered known tserver with Master: 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287)
I20251026 21:07:10.611390 11133 ts_manager.cc:194] Registered new tserver with Master: 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287)
I20251026 21:07:10.611480 10521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:58711
I20251026 21:07:10.611981 11317 ts_manager.cc:194] Registered new tserver with Master: 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287)
I20251026 21:07:10.612998 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.596041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.3.175.61:34739)
1026 21:07:10.596518 (+ 477us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.596523 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.596536 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:10.596575 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.596578 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.596634 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.597737 (+ 1103us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.597841 (+ 104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.599870 (+ 2029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.602800 (+ 2930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.602992 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.603090 (+ 98us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.603117 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.603120 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.603139 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.611257 (+ 8118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.611764 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.611771 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.611773 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.611830 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.612112 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.612117 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.612119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.612332 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.612469 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.612676 (+ 207us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.612799 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":64,"threads_started":1}
I20251026 21:07:10.615213 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.595689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.3.175.60:36199)
1026 21:07:10.597107 (+ 1418us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.597112 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.598382 (+ 1270us) server_negotiation.cc:408] Connection header received
1026 21:07:10.598482 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.598485 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.598544 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.598644 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.601873 (+ 3229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.602665 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.604245 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.604456 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.607694 (+ 3238us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.607712 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.607715 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.607742 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.613447 (+ 5705us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.614064 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.614067 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.614068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.614119 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.614370 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.614372 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.614374 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.614558 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.614705 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.614911 (+ 206us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.615042 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20251026 21:07:10.615231 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.594857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.3.175.62:41917)
1026 21:07:10.598737 (+ 3880us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.598741 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.598754 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:10.598788 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.598791 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.598838 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.599037 (+ 199us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.600935 (+ 1898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.605122 (+ 4187us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.607701 (+ 2579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.608197 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.610131 (+ 1934us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.610199 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.610203 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.610231 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.613774 (+ 3543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.614204 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.614206 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.614208 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.614247 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.614469 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.614471 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.614473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.614637 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.614752 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.614920 (+ 168us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.615041 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3780,"thread_start_us":61,"threads_started":1}
I20251026 21:07:10.617755 10521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:07:10.620169 10521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:10.637200 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:40987 (local address 127.0.0.1:42624)
1026 21:07:10.633712 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.633726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.633822 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.634640 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.634643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.634662 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:10.634943 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.634954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.636112 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.636115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.636887 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.636895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.636984 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.637003 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.637040 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.637084 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20251026 21:07:10.637732 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:42287 (local address 127.0.0.1:55746)
1026 21:07:10.634321 (+ 858us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.634334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.634418 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.634837 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.634840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.634859 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:10.635102 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.635110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.636565 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.636567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.637371 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.637381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.637489 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.637505 (+ 16us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.637555 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.637613 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":78,"threads_started":1}
I20251026 21:07:10.638062 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.3.175.1:40987)
1026 21:07:10.634152 (+ 298us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.634157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.634178 (+ 21us) server_negotiation.cc:408] Connection header received
1026 21:07:10.634267 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.634272 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.634509 (+ 237us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.634660 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:10.635104 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.636008 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.637007 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.637559 (+ 552us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.637700 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.637769 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.637882 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20251026 21:07:10.638669 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.3.175.3:42287)
1026 21:07:10.634194 (+ 593us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.634201 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.634511 (+ 310us) server_negotiation.cc:408] Connection header received
1026 21:07:10.634585 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.634589 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.634718 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.634868 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:10.635365 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.636459 (+ 1094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.637622 (+ 1163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.638273 (+ 651us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.638373 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.638448 (+ 75us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.638547 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":130,"threads_started":1}
I20251026 21:07:10.639853 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41823 (local address 127.0.0.1:46306)
1026 21:07:10.633858 (+ 729us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.633885 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.633969 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.635885 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.635889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.635907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:10.636133 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.636141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.638552 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.638555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.639468 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.639479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.639602 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.639619 (+ 17us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.639668 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.639722 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":110,"threads_started":1}
I20251026 21:07:10.640695 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.633304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.3.175.2:41823)
1026 21:07:10.635349 (+ 2045us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.635355 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.635486 (+ 131us) server_negotiation.cc:408] Connection header received
1026 21:07:10.635573 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.635577 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.635731 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.635895 (+ 164us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:10.637381 (+ 1486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.638422 (+ 1041us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.639815 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.640332 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.640428 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.640487 (+ 59us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.640569 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1917,"thread_start_us":100,"threads_started":1}
I20251026 21:07:10.641274 11718 tablet_service.cc:1505] Processing CreateTablet for tablet ac137f6440d344b5b77c0ba3e07a728d (DEFAULT_TABLE table=test-table [id=251131c881004084bcf87f28c6850ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:10.641265 11446 tablet_service.cc:1505] Processing CreateTablet for tablet ac137f6440d344b5b77c0ba3e07a728d (DEFAULT_TABLE table=test-table [id=251131c881004084bcf87f28c6850ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:10.642117 11718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac137f6440d344b5b77c0ba3e07a728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:10.642165 11446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac137f6440d344b5b77c0ba3e07a728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:10.643244 11582 tablet_service.cc:1505] Processing CreateTablet for tablet ac137f6440d344b5b77c0ba3e07a728d (DEFAULT_TABLE table=test-table [id=251131c881004084bcf87f28c6850ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:10.644158 11582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac137f6440d344b5b77c0ba3e07a728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:10.648645 11818 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Bootstrap starting.
I20251026 21:07:10.649816 11819 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Bootstrap starting.
I20251026 21:07:10.651082 11818 tablet_bootstrap.cc:654] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:10.651160 11820 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Bootstrap starting.
I20251026 21:07:10.651839 11819 tablet_bootstrap.cc:654] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:10.652307 11818 log.cc:826] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:10.652613 11819 log.cc:826] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:10.653560 11820 tablet_bootstrap.cc:654] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:10.654310 11820 log.cc:826] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:10.654583 11819 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: No bootstrap required, opened a new log
I20251026 21:07:10.654795 11819 ts_tablet_manager.cc:1403] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 21:07:10.656214 11818 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: No bootstrap required, opened a new log
I20251026 21:07:10.656216 11820 tablet_bootstrap.cc:492] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: No bootstrap required, opened a new log
I20251026 21:07:10.656389 11820 ts_tablet_manager.cc:1403] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:07:10.656400 11818 ts_tablet_manager.cc:1403] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251026 21:07:10.658084 11819 raft_consensus.cc:359] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.658320 11819 raft_consensus.cc:385] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:10.658370 11819 raft_consensus.cc:740] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643943dcd2704d72852a37536e03a82f, State: Initialized, Role: FOLLOWER
I20251026 21:07:10.658859 11819 consensus_queue.cc:260] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.659317 11818 raft_consensus.cc:359] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.659348 11820 raft_consensus.cc:359] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.659518 11820 raft_consensus.cc:385] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:10.659513 11818 raft_consensus.cc:385] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:10.659560 11820 raft_consensus.cc:740] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226478220f964d02baead480f14a9c72, State: Initialized, Role: FOLLOWER
I20251026 21:07:10.659569 11818 raft_consensus.cc:740] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0950373eb54a7886e1314b21421055, State: Initialized, Role: FOLLOWER
I20251026 21:07:10.659854 11819 ts_tablet_manager.cc:1434] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251026 21:07:10.660013 11818 consensus_queue.cc:260] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.660617 11513 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
I20251026 21:07:10.660831 11818 ts_tablet_manager.cc:1434] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20251026 21:07:10.661012 11785 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
I20251026 21:07:10.660012 11820 consensus_queue.cc:260] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.663414 11649 heartbeater.cc:499] Master 127.3.175.62:41917 was elected leader, sending a full tablet report...
I20251026 21:07:10.664232 11820 ts_tablet_manager.cc:1434] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20251026 21:07:10.671571 11515 tablet.cc:2378] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:07:10.840956 11787 tablet.cc:2378] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:07:10.861600 11825 raft_consensus.cc:493] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:10.861793 11825 raft_consensus.cc:515] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.862963 11825 leader_election.cc:290] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987), 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:10.865648 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.863244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:40987 (local address 127.3.175.3:36537)
1026 21:07:10.863390 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.863408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.863512 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.863788 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.863791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.863808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:10.864020 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.864026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.864713 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.864716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.865276 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.865283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.865389 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.865411 (+ 22us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.865463 (+ 52us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.865519 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20251026 21:07:10.865823 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.863250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41823 (local address 127.3.175.3:56841)
1026 21:07:10.863541 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:10.863555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:10.863664 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:10.863919 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:10.863922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:10.863939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:10.864136 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.864142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.864832 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.864835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:10.865407 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:10.865414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.865518 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.865536 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:07:10.865585 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 21:07:10.865637 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":151}
I20251026 21:07:10.866123 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.863341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:36537 (local address 127.3.175.1:40987)
1026 21:07:10.863476 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.863481 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.863496 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:10.863645 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.863648 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.863692 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.863764 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:10.864139 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.864605 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.865388 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.865845 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.865902 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.865957 (+ 55us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.866018 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 21:07:10.866292 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.863382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:56841 (local address 127.3.175.2:41823)
1026 21:07:10.863515 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.863520 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.863615 (+ 95us) server_negotiation.cc:408] Connection header received
1026 21:07:10.863773 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.863776 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.863818 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.863908 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:10.864253 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.864713 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.865521 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.866006 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.866053 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.866104 (+ 51us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.866146 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 21:07:10.866688 11466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac137f6440d344b5b77c0ba3e07a728d" candidate_uuid: "8b0950373eb54a7886e1314b21421055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643943dcd2704d72852a37536e03a82f" is_pre_election: true
I20251026 21:07:10.866832 11602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac137f6440d344b5b77c0ba3e07a728d" candidate_uuid: "8b0950373eb54a7886e1314b21421055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226478220f964d02baead480f14a9c72" is_pre_election: true
I20251026 21:07:10.866962 11466 raft_consensus.cc:2468] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0950373eb54a7886e1314b21421055 in term 0.
I20251026 21:07:10.867094 11602 raft_consensus.cc:2468] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0950373eb54a7886e1314b21421055 in term 0.
I20251026 21:07:10.867445 11671 leader_election.cc:304] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643943dcd2704d72852a37536e03a82f, 8b0950373eb54a7886e1314b21421055; no voters:
I20251026 21:07:10.867686 11825 raft_consensus.cc:2804] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:10.867756 11825 raft_consensus.cc:493] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:10.867802 11825 raft_consensus.cc:3060] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:10.868806 11825 raft_consensus.cc:515] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.869220 11825 leader_election.cc:290] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [CANDIDATE]: Term 1 election: Requested vote from peers 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987), 226478220f964d02baead480f14a9c72 (127.3.175.2:41823)
I20251026 21:07:10.869599 11602 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac137f6440d344b5b77c0ba3e07a728d" candidate_uuid: "8b0950373eb54a7886e1314b21421055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226478220f964d02baead480f14a9c72"
I20251026 21:07:10.869590 11466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac137f6440d344b5b77c0ba3e07a728d" candidate_uuid: "8b0950373eb54a7886e1314b21421055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643943dcd2704d72852a37536e03a82f"
I20251026 21:07:10.869719 11602 raft_consensus.cc:3060] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:10.869719 11466 raft_consensus.cc:3060] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:10.870937 11466 raft_consensus.cc:2468] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0950373eb54a7886e1314b21421055 in term 1.
I20251026 21:07:10.870937 11602 raft_consensus.cc:2468] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0950373eb54a7886e1314b21421055 in term 1.
I20251026 21:07:10.871320 11671 leader_election.cc:304] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643943dcd2704d72852a37536e03a82f, 8b0950373eb54a7886e1314b21421055; no voters:
I20251026 21:07:10.871554 11825 raft_consensus.cc:2804] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:10.871771 11825 raft_consensus.cc:697] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0950373eb54a7886e1314b21421055, State: Running, Role: LEADER
I20251026 21:07:10.872241 11825 consensus_queue.cc:237] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } }
I20251026 21:07:10.875989 10519 catalog_manager.cc:5649] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0950373eb54a7886e1314b21421055 (127.3.175.3). New cstate: current_term: 1 leader_uuid: "8b0950373eb54a7886e1314b21421055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } health_report { overall_health: HEALTHY } } }
W20251026 21:07:10.882097 11651 tablet.cc:2378] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:07:10.935487 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.932617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.3.175.3:42287)
1026 21:07:10.932747 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.932750 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.932764 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:10.932862 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.932865 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.932919 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.933011 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:10.933328 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.933804 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.934442 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.934578 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.934671 (+ 93us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:10.935109 (+ 438us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:10.935239 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.935318 (+ 79us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.935374 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:07:10.941270 11602 raft_consensus.cc:1275] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Refusing update from remote peer 8b0950373eb54a7886e1314b21421055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:10.941270 11466 raft_consensus.cc:1275] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Refusing update from remote peer 8b0950373eb54a7886e1314b21421055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:10.941928 11825 consensus_queue.cc:1048] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:10.942344 11827 consensus_queue.cc:1048] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:10.950201 11830 mvcc.cc:204] Tried to move back new op lower bound from 7215156555527442432 to 7215156555256004608. Current Snapshot: MvccSnapshot[applied={T|T < 7215156555527442432}]
I20251026 21:07:10.964573 10519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:10.964776 10519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:10.969153 10519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 1c2e62502e21436a81044f938290c8f2: Sending DeleteTablet for 3 replicas of tablet ac137f6440d344b5b77c0ba3e07a728d
I20251026 21:07:10.969935 11446 tablet_service.cc:1552] Processing DeleteTablet for tablet ac137f6440d344b5b77c0ba3e07a728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:10 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:42624
I20251026 21:07:10.970157 11718 tablet_service.cc:1552] Processing DeleteTablet for tablet ac137f6440d344b5b77c0ba3e07a728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:10 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:55746
I20251026 21:07:10.970180 11582 tablet_service.cc:1552] Processing DeleteTablet for tablet ac137f6440d344b5b77c0ba3e07a728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:10 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46306
I20251026 21:07:10.971030 11840 tablet_replica.cc:333] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: stopping tablet replica
I20251026 21:07:10.971366 11840 raft_consensus.cc:2243] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:10.971730 11840 raft_consensus.cc:2272] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:10.972134 11841 tablet_replica.cc:333] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: stopping tablet replica
I20251026 21:07:10.972412 11841 raft_consensus.cc:2243] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:10.972615 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10489
I20251026 21:07:10.973459 11841 raft_consensus.cc:2272] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:10.974735 11840 ts_tablet_manager.cc:1916] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:10.975544 11842 tablet_replica.cc:333] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: stopping tablet replica
I20251026 21:07:10.975871 11842 raft_consensus.cc:2243] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:07:10.976401 11842 raft_consensus.cc:2272] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:10.977798 11842 ts_tablet_manager.cc:1916] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:10.978083 11841 ts_tablet_manager.cc:1916] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:10.980966 11842 ts_tablet_manager.cc:1929] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:10.981133 11842 log.cc:1199] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/ac137f6440d344b5b77c0ba3e07a728d
I20251026 21:07:10.981209 11841 ts_tablet_manager.cc:1929] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:10.981321 11841 log.cc:1199] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/ac137f6440d344b5b77c0ba3e07a728d
I20251026 21:07:10.982339 11842 ts_tablet_manager.cc:1950] T ac137f6440d344b5b77c0ba3e07a728d P 8b0950373eb54a7886e1314b21421055: Deleting consensus metadata
I20251026 21:07:10.983250 11840 ts_tablet_manager.cc:1929] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:10.983376 11840 log.cc:1199] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/ac137f6440d344b5b77c0ba3e07a728d
W20251026 21:07:10.983603 11674 connection.cc:441] server connection from 127.0.0.1:55746 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55746 (request call id 1) could send its response
I20251026 21:07:10.983752 11840 ts_tablet_manager.cc:1950] T ac137f6440d344b5b77c0ba3e07a728d P 643943dcd2704d72852a37536e03a82f: Deleting consensus metadata
I20251026 21:07:10.981689 11841 ts_tablet_manager.cc:1950] T ac137f6440d344b5b77c0ba3e07a728d P 226478220f964d02baead480f14a9c72: Deleting consensus metadata
W20251026 21:07:10.984882 11401 connection.cc:441] server connection from 127.0.0.1:42624 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42624 (request call id 1) could send its response
W20251026 21:07:10.985915 11538 connection.cc:441] server connection from 127.0.0.1:46306 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46306 (request call id 1) could send its response
I20251026 21:07:10.994794 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.984933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.3.175.60:36199)
1026 21:07:10.985071 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.985078 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.986101 (+ 1023us) server_negotiation.cc:408] Connection header received
1026 21:07:10.986294 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.986298 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.986358 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.986446 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.987595 (+ 1149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.988390 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.989519 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.989730 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.990935 (+ 1205us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.990955 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.990959 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.990987 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.992915 (+ 1928us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.993365 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.993368 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.993370 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.993415 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.993634 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.993637 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.993639 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.993805 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.993988 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.994527 (+ 539us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.994637 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:07:10.994796 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:10.984413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.3.175.61:34739)
1026 21:07:10.984540 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:10.984545 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:10.985680 (+ 1135us) server_negotiation.cc:408] Connection header received
1026 21:07:10.985807 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:10.985810 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:10.985886 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:10.985979 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:10.987593 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.988176 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:10.989188 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:10.989389 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:10.991064 (+ 1675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:10.991087 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:10.991090 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:10.991155 (+ 65us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:10.993085 (+ 1930us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.993549 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.993553 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.993555 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.993603 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:10.993854 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:10.993857 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:10.993859 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:10.994061 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:10.994169 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:10.994495 (+ 326us) server_negotiation.cc:300] Negotiation successful
1026 21:07:10.994629 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251026 21:07:11.953747 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:11.953310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:44071)
1026 21:07:11.953577 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:11.953646 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
W20251026 21:07:11.954035 11649 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:41917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:11.956545 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:11.956133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:47805)
1026 21:07:11.956386 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:11.956461 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
W20251026 21:07:11.956825 11785 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:41917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:11.959193 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:11.958702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:48767)
1026 21:07:11.958992 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:11.959068 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":70,"threads_started":1}
W20251026 21:07:11.959421 11513 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:41917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.469386 11853 raft_consensus.cc:493] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2e62502e21436a81044f938290c8f2)
I20251026 21:07:12.469614 11853 raft_consensus.cc:515] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:12.470144 11854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2e62502e21436a81044f938290c8f2)
I20251026 21:07:12.470255 11854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:12.470798 11853 leader_election.cc:290] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61:34739)
I20251026 21:07:12.470777 11854 leader_election.cc:290] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199)
I20251026 21:07:12.471484 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.470967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50486)
1026 21:07:12.471272 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.471376 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":60,"threads_started":1}
I20251026 21:07:12.471446 11327 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02b06b2ce5944ee9b05df06891a13b35" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "329609e082a84ad3bde9e588044d26c4" is_pre_election: true
I20251026 21:07:12.471508 11135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "329609e082a84ad3bde9e588044d26c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02b06b2ce5944ee9b05df06891a13b35" is_pre_election: true
I20251026 21:07:12.471670 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.470992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50488)
1026 21:07:12.471422 (+ 430us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.471495 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":150,"threads_started":1}
W20251026 21:07:12.471685 11304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.471735 11327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b06b2ce5944ee9b05df06891a13b35 in term 1.
I20251026 21:07:12.471743 11135 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 329609e082a84ad3bde9e588044d26c4 in term 1.
W20251026 21:07:12.471975 11120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.472150 11118 leader_election.cc:304] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35, 329609e082a84ad3bde9e588044d26c4; no voters:
I20251026 21:07:12.472333 11854 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 21:07:12.472389 11854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c2e62502e21436a81044f938290c8f2)
I20251026 21:07:12.472429 11854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 1 FOLLOWER]: Advancing to term 2
I20251026 21:07:12.472434 11301 leader_election.cc:304] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35, 329609e082a84ad3bde9e588044d26c4; no voters:
I20251026 21:07:12.472645 11853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 21:07:12.472697 11853 raft_consensus.cc:493] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c2e62502e21436a81044f938290c8f2)
I20251026 21:07:12.472760 11853 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 21:07:12.472783 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.472477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50508)
1026 21:07:12.472629 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.472684 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20251026 21:07:12.472878 11304 leader_election.cc:336] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.472985 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.472773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50524)
1026 21:07:12.472872 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.472918 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20251026 21:07:12.473142 11120 leader_election.cc:336] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.473737 11854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:12.473942 11853 raft_consensus.cc:515] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:12.474170 11854 leader_election.cc:290] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 election: Requested vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199)
I20251026 21:07:12.474320 11853 leader_election.cc:290] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 election: Requested vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 02b06b2ce5944ee9b05df06891a13b35 (127.3.175.61:34739)
I20251026 21:07:12.474486 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.474268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50532)
1026 21:07:12.474380 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.474424 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20251026 21:07:12.474648 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.474416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50546)
1026 21:07:12.474530 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.474586 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20251026 21:07:12.474677 11327 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02b06b2ce5944ee9b05df06891a13b35" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "329609e082a84ad3bde9e588044d26c4"
I20251026 21:07:12.474836 11327 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02b06b2ce5944ee9b05df06891a13b35 in current term 2: Already voted for candidate 329609e082a84ad3bde9e588044d26c4 in this term.
I20251026 21:07:12.474869 11135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "329609e082a84ad3bde9e588044d26c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02b06b2ce5944ee9b05df06891a13b35"
I20251026 21:07:12.475030 11135 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 329609e082a84ad3bde9e588044d26c4 in current term 2: Already voted for candidate 02b06b2ce5944ee9b05df06891a13b35 in this term.
I20251026 21:07:12.475486 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.475239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50554)
1026 21:07:12.475377 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.475429 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20251026 21:07:12.475486 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.475314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50568)
1026 21:07:12.475394 (+ 80us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.475436 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20251026 21:07:12.475581 11304 leader_election.cc:336] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
W20251026 21:07:12.475625 11120 leader_election.cc:336] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:12.475648 11304 leader_election.cc:304] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 329609e082a84ad3bde9e588044d26c4; no voters: 02b06b2ce5944ee9b05df06891a13b35, 1c2e62502e21436a81044f938290c8f2
I20251026 21:07:12.475695 11120 leader_election.cc:304] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35; no voters: 1c2e62502e21436a81044f938290c8f2, 329609e082a84ad3bde9e588044d26c4
I20251026 21:07:12.475775 11853 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 21:07:12.475842 11854 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 21:07:12.955058 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.954609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:34581)
1026 21:07:12.954898 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.954962 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20251026 21:07:12.957916 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.957360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:32863)
1026 21:07:12.957722 (+ 362us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.957793 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20251026 21:07:12.960436 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:12.960024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:52343)
1026 21:07:12.960279 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:12.960345 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20251026 21:07:13.956452 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:13.956041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:43369)
1026 21:07:13.956299 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:13.956357 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20251026 21:07:13.959066 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:13.958564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:48061)
1026 21:07:13.958861 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:13.958932 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20251026 21:07:13.961697 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:13.961124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:59165)
1026 21:07:13.961528 (+ 404us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:13.961581 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":76,"threads_started":1}
I20251026 21:07:14.226094 11865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:14.226265 11865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:14.226698 11865 leader_election.cc:290] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199)
I20251026 21:07:14.227205 11327 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02b06b2ce5944ee9b05df06891a13b35" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "329609e082a84ad3bde9e588044d26c4" is_pre_election: true
I20251026 21:07:14.227346 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.226899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50802)
1026 21:07:14.227155 (+ 256us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.227211 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20251026 21:07:14.227382 11327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b06b2ce5944ee9b05df06891a13b35 in term 2.
I20251026 21:07:14.227725 11118 leader_election.cc:304] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35, 329609e082a84ad3bde9e588044d26c4; no voters:
I20251026 21:07:14.227910 11865 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 21:07:14.227974 11865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:14.228021 11865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 2 FOLLOWER]: Advancing to term 3
I20251026 21:07:14.228304 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.227965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50818)
1026 21:07:14.228187 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.228245 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20251026 21:07:14.228420 11120 leader_election.cc:336] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:14.229076 11865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:14.229411 11865 leader_election.cc:290] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 election: Requested vote from peers 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917), 329609e082a84ad3bde9e588044d26c4 (127.3.175.60:36199)
I20251026 21:07:14.229746 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.229523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50822)
1026 21:07:14.229633 (+ 110us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.229678 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251026 21:07:14.229818 11327 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02b06b2ce5944ee9b05df06891a13b35" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "329609e082a84ad3bde9e588044d26c4"
I20251026 21:07:14.230211 11327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 2 FOLLOWER]: Advancing to term 3
I20251026 21:07:14.230579 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.230282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50834)
1026 21:07:14.230469 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.230523 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251026 21:07:14.230682 11120 leader_election.cc:336] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
I20251026 21:07:14.231375 11327 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b06b2ce5944ee9b05df06891a13b35 in term 3.
I20251026 21:07:14.231752 11118 leader_election.cc:304] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02b06b2ce5944ee9b05df06891a13b35, 329609e082a84ad3bde9e588044d26c4; no voters: 1c2e62502e21436a81044f938290c8f2
I20251026 21:07:14.232002 11865 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 21:07:14.232196 11865 raft_consensus.cc:697] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [term 3 LEADER]: Becoming Leader. State: Replica: 02b06b2ce5944ee9b05df06891a13b35, State: Running, Role: LEADER
I20251026 21:07:14.232455 11865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } }
I20251026 21:07:14.233600 11868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b06b2ce5944ee9b05df06891a13b35. Latest consensus state: current_term: 3 leader_uuid: "02b06b2ce5944ee9b05df06891a13b35" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:14.233691 11868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:14.233989 11870 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:14.236156 11870 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:14.236546 11870 catalog_manager.cc:1269] Loaded cluster ID: a744a34c9b134fdc866f41e652cc9d9b
I20251026 21:07:14.236600 11870 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:14.237149 11870 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:14.237522 11870 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Loaded TSK: 0
I20251026 21:07:14.238232 11870 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:07:14.281440 11133 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:07:14.283968 11133 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:14.286986 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.286658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50858)
1026 21:07:14.286807 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.286898 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20251026 21:07:14.287089 11327 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [term 3 FOLLOWER]: Refusing update from remote peer 02b06b2ce5944ee9b05df06891a13b35: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251026 21:07:14.287673 11868 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251026 21:07:14.287777 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.287514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50862)
1026 21:07:14.287635 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.287681 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251026 21:07:14.287923 11120 consensus_peers.cc:597] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 -> Peer 1c2e62502e21436a81044f938290c8f2 (127.3.175.62:41917): Couldn't send request to peer 1c2e62502e21436a81044f938290c8f2. Status: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 21:07:14.289209 11872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b06b2ce5944ee9b05df06891a13b35. Latest consensus state: current_term: 3 leader_uuid: "02b06b2ce5944ee9b05df06891a13b35" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:14.289340 11872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:14.290438 11868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "02b06b2ce5944ee9b05df06891a13b35" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:14.290561 11868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:14.291076 11872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "02b06b2ce5944ee9b05df06891a13b35" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1c2e62502e21436a81044f938290c8f2" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 41917 } } peers { permanent_uuid: "02b06b2ce5944ee9b05df06891a13b35" member_type: VOTER last_known_addr { host: "127.3.175.61" port: 34739 } attrs { promote: false } } peers { permanent_uuid: "329609e082a84ad3bde9e588044d26c4" member_type: VOTER last_known_addr { host: "127.3.175.60" port: 36199 } attrs { promote: false } } }
I20251026 21:07:14.291174 11872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329609e082a84ad3bde9e588044d26c4 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 21:07:14.300663 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.296892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:42287 (local address 127.0.0.1:55764)
1026 21:07:14.297016 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.297033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:14.297140 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:14.297672 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:14.297686 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:14.297710 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:14.297999 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.298009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.299536 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.299540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:14.300236 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.300248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.300359 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.300382 (+ 23us) client_negotiation.cc:769] Sending connection context
1026 21:07:14.300432 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 21:07:14.300509 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251026 21:07:14.301219 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.297765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:40987 (local address 127.0.0.1:42628)
1026 21:07:14.298066 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.298081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:14.298159 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:14.298743 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:14.298746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:14.298764 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:14.298993 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.299000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.300011 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.300014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:14.300828 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.300840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.300944 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.300963 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:07:14.301012 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 21:07:14.301069 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20251026 21:07:14.301569 11877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.297085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.3.175.3:42287)
1026 21:07:14.297389 (+ 304us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.297393 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.297409 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:07:14.297463 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.297469 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.297538 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.298455 (+ 917us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:14.298581 (+ 126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.299412 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.300362 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.301109 (+ 747us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.301160 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.301212 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.301296 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20251026 21:07:14.301932 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.297420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:41823 (local address 127.0.0.1:46312)
1026 21:07:14.297998 (+ 578us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.298010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:14.298100 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:14.298358 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:14.298362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:14.298378 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:14.298580 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.298586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.300849 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.300853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:14.301575 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.301583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.301696 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.301712 (+ 16us) client_negotiation.cc:769] Sending connection context
1026 21:07:14.301758 (+ 46us) client_negotiation.cc:240] Negotiation successful
1026 21:07:14.301816 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":100,"threads_started":1}
I20251026 21:07:14.302110 11718 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d (DEFAULT_TABLE table=test-table [id=85e5a355b457490faafaa745a4a87ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:14.302275 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.298275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.3.175.1:40987)
1026 21:07:14.298509 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.298513 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.298527 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:14.298580 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.298583 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.298630 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.298733 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:14.299120 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.299765 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.301430 (+ 1665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.301974 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.302009 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.302062 (+ 53us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.302130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20251026 21:07:14.302451 11718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad4ae3222994d04a6215f3753e81b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:14.302634 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.297587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.3.175.2:41823)
1026 21:07:14.297846 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.297849 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.298080 (+ 231us) server_negotiation.cc:408] Connection header received
1026 21:07:14.298208 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.298212 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.298261 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.298331 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:14.298690 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.299424 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.301811 (+ 2387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.302333 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.302378 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.302429 (+ 51us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.302504 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20251026 21:07:14.302923 11446 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d (DEFAULT_TABLE table=test-table [id=85e5a355b457490faafaa745a4a87ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:14.303171 11446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad4ae3222994d04a6215f3753e81b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:14.304931 11882 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Bootstrap starting.
I20251026 21:07:14.305550 11883 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: Bootstrap starting.
I20251026 21:07:14.305760 11582 tablet_service.cc:1505] Processing CreateTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d (DEFAULT_TABLE table=test-table [id=85e5a355b457490faafaa745a4a87ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:14.306073 11582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad4ae3222994d04a6215f3753e81b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:14.306124 11882 tablet_bootstrap.cc:654] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:14.306558 11883 tablet_bootstrap.cc:654] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:14.307729 11883 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: No bootstrap required, opened a new log
I20251026 21:07:14.307824 11883 ts_tablet_manager.cc:1403] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:14.308157 11882 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: No bootstrap required, opened a new log
I20251026 21:07:14.308251 11882 ts_tablet_manager.cc:1403] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:14.308597 11883 raft_consensus.cc:359] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.308712 11883 raft_consensus.cc:385] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:14.308725 11886 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Bootstrap starting.
I20251026 21:07:14.308753 11883 raft_consensus.cc:740] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643943dcd2704d72852a37536e03a82f, State: Initialized, Role: FOLLOWER
I20251026 21:07:14.308749 11882 raft_consensus.cc:359] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.308848 11882 raft_consensus.cc:385] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:14.308878 11882 raft_consensus.cc:740] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0950373eb54a7886e1314b21421055, State: Initialized, Role: FOLLOWER
I20251026 21:07:14.308919 11883 consensus_queue.cc:260] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.309036 11882 consensus_queue.cc:260] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.309299 11883 ts_tablet_manager.cc:1434] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 21:07:14.309433 11882 ts_tablet_manager.cc:1434] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:14.310132 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.309798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:54025)
1026 21:07:14.309977 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.310040 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20251026 21:07:14.310642 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.310419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:32869)
1026 21:07:14.310522 (+ 103us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.310574 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20251026 21:07:14.311334 11886 tablet_bootstrap.cc:654] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:14.312666 11886 tablet_bootstrap.cc:492] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: No bootstrap required, opened a new log
I20251026 21:07:14.312772 11886 ts_tablet_manager.cc:1403] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251026 21:07:14.313659 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.310594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:47037)
1026 21:07:14.313520 (+ 2926us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.313580 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2817}
I20251026 21:07:14.313879 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.313583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:46127)
1026 21:07:14.313743 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.313797 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20251026 21:07:14.314007 11886 raft_consensus.cc:359] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.314100 11886 raft_consensus.cc:385] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:14.314128 11886 raft_consensus.cc:740] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226478220f964d02baead480f14a9c72, State: Initialized, Role: FOLLOWER
I20251026 21:07:14.314258 11886 consensus_queue.cc:260] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.314636 11886 ts_tablet_manager.cc:1434] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:14.315272 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.314984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:44451)
1026 21:07:14.315137 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.315187 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20251026 21:07:14.315326 11514 heartbeater.cc:499] Master 127.3.175.61:34739 was elected leader, sending a full tablet report...
I20251026 21:07:14.316887 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.315942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:46643)
1026 21:07:14.316763 (+ 821us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.316812 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756}
I20251026 21:07:14.649333 11890 raft_consensus.cc:493] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:14.649497 11890 raft_consensus.cc:515] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.650692 11890 leader_election.cc:290] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287), 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
I20251026 21:07:14.654597 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.650998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:40987 (local address 127.3.175.2:56417)
1026 21:07:14.651217 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.651235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:14.651382 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:14.651732 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:14.651735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:14.651763 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:14.652097 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.652105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.653167 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.653170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:14.654143 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.654153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.654272 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.654295 (+ 23us) client_negotiation.cc:769] Sending connection context
1026 21:07:14.654350 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 21:07:14.654414 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"spinlock_wait_cycles":9344}
I20251026 21:07:14.654597 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.651004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:42287 (local address 127.3.175.2:45109)
1026 21:07:14.651370 (+ 366us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.651382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:14.651464 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:14.651732 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:14.651735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:14.651769 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:14.652097 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.652105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.653167 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.653170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:14.654143 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:14.654153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.654272 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.654296 (+ 24us) client_negotiation.cc:769] Sending connection context
1026 21:07:14.654350 (+ 54us) client_negotiation.cc:240] Negotiation successful
1026 21:07:14.654413 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":3,"spinlock_wait_cycles":49664,"thread_start_us":131,"threads_started":1}
I20251026 21:07:14.655017 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.651158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:56417 (local address 127.3.175.1:40987)
1026 21:07:14.651422 (+ 264us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.651429 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.651470 (+ 41us) server_negotiation.cc:408] Connection header received
1026 21:07:14.651537 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.651540 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.651601 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.651698 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:14.652256 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.653039 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.654275 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.654754 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.654794 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.654846 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.654902 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20251026 21:07:14.655017 11877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.651104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:45109 (local address 127.3.175.3:42287)
1026 21:07:14.651303 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.651309 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.651462 (+ 153us) server_negotiation.cc:408] Connection header received
1026 21:07:14.651549 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.651553 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.651618 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.651705 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:14.652256 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.653039 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.654275 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.654754 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.654794 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.654846 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.654909 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251026 21:07:14.655479 11466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ad4ae3222994d04a6215f3753e81b0d" candidate_uuid: "226478220f964d02baead480f14a9c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643943dcd2704d72852a37536e03a82f" is_pre_election: true
I20251026 21:07:14.655555 11738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ad4ae3222994d04a6215f3753e81b0d" candidate_uuid: "226478220f964d02baead480f14a9c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0950373eb54a7886e1314b21421055" is_pre_election: true
I20251026 21:07:14.655638 11466 raft_consensus.cc:2468] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226478220f964d02baead480f14a9c72 in term 0.
I20251026 21:07:14.655771 11738 raft_consensus.cc:2468] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226478220f964d02baead480f14a9c72 in term 0.
I20251026 21:07:14.656091 11535 leader_election.cc:304] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226478220f964d02baead480f14a9c72, 643943dcd2704d72852a37536e03a82f; no voters:
I20251026 21:07:14.656340 11890 raft_consensus.cc:2804] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:14.656407 11890 raft_consensus.cc:493] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:14.656452 11890 raft_consensus.cc:3060] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:14.657367 11890 raft_consensus.cc:515] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.657711 11890 leader_election.cc:290] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287), 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987)
I20251026 21:07:14.658048 11738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ad4ae3222994d04a6215f3753e81b0d" candidate_uuid: "226478220f964d02baead480f14a9c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0950373eb54a7886e1314b21421055"
I20251026 21:07:14.658104 11466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ad4ae3222994d04a6215f3753e81b0d" candidate_uuid: "226478220f964d02baead480f14a9c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643943dcd2704d72852a37536e03a82f"
I20251026 21:07:14.658164 11738 raft_consensus.cc:3060] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:14.658236 11466 raft_consensus.cc:3060] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:14.659034 11466 raft_consensus.cc:2468] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226478220f964d02baead480f14a9c72 in term 1.
I20251026 21:07:14.659071 11738 raft_consensus.cc:2468] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226478220f964d02baead480f14a9c72 in term 1.
I20251026 21:07:14.659390 11536 leader_election.cc:304] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226478220f964d02baead480f14a9c72, 8b0950373eb54a7886e1314b21421055; no voters:
I20251026 21:07:14.659574 11890 raft_consensus.cc:2804] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:14.659811 11890 raft_consensus.cc:697] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 1 LEADER]: Becoming Leader. State: Replica: 226478220f964d02baead480f14a9c72, State: Running, Role: LEADER
I20251026 21:07:14.660079 11890 consensus_queue.cc:237] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } }
I20251026 21:07:14.660147 11650 heartbeater.cc:499] Master 127.3.175.61:34739 was elected leader, sending a full tablet report...
I20251026 21:07:14.660543 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.660287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:59825)
1026 21:07:14.660410 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.660468 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20251026 21:07:14.662465 11132 catalog_manager.cc:5649] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 226478220f964d02baead480f14a9c72 (127.3.175.2). New cstate: current_term: 1 leader_uuid: "226478220f964d02baead480f14a9c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226478220f964d02baead480f14a9c72" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 41823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 } health_report { overall_health: UNKNOWN } } }
I20251026 21:07:14.693462 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.693181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50864)
1026 21:07:14.693312 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.693376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20251026 21:07:14.694170 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.693971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.0.0.1:50880)
1026 21:07:14.694077 (+ 106us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.694116 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 21:07:14.756584 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.753482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.3.175.2:41823)
1026 21:07:14.753656 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:14.753663 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:14.753680 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:07:14.753760 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:14.753763 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:14.753818 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:14.753908 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:14.754261 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.754741 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:14.755435 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:14.755627 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:14.755681 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:14.756195 (+ 514us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:14.756304 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:14.756392 (+ 88us) server_negotiation.cc:300] Negotiation successful
1026 21:07:14.756443 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251026 21:07:14.759220 11466 raft_consensus.cc:1275] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Refusing update from remote peer 226478220f964d02baead480f14a9c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:14.759495 11738 raft_consensus.cc:1275] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Refusing update from remote peer 226478220f964d02baead480f14a9c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:14.759783 11892 consensus_queue.cc:1048] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643943dcd2704d72852a37536e03a82f" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:14.760005 11890 consensus_queue.cc:1048] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0950373eb54a7886e1314b21421055" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:14.761845 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.761384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:43053)
1026 21:07:14.761677 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.761750 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20251026 21:07:14.762018 11786 heartbeater.cc:499] Master 127.3.175.61:34739 was elected leader, sending a full tablet report...
I20251026 21:07:14.762574 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.762273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:46559)
1026 21:07:14.762446 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.762498 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20251026 21:07:14.765434 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.765187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:39351)
1026 21:07:14.765299 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.765357 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 21:07:14.766515 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.766075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.2:40467)
1026 21:07:14.766394 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.766444 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20251026 21:07:14.768741 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.768276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.1:60891)
1026 21:07:14.768578 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.768634 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20251026 21:07:14.769297 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:14.768998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:41917 (local address 127.3.175.3:43151)
1026 21:07:14.769168 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.769220 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20251026 21:07:14.774091 11133 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:14.774245 11133 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:14.778474 11133 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 02b06b2ce5944ee9b05df06891a13b35: Sending DeleteTablet for 3 replicas of tablet 7ad4ae3222994d04a6215f3753e81b0d
I20251026 21:07:14.779198 11582 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:14 UTC) from {username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:46312
I20251026 21:07:14.779316 11718 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:14 UTC) from {username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:55764
I20251026 21:07:14.779369 11446 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ad4ae3222994d04a6215f3753e81b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:14 UTC) from {username='oryx', principal='oryx/127.3.175.61@KRBTEST.COM'} at 127.0.0.1:42628
I20251026 21:07:14.779701 11907 tablet_replica.cc:333] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: stopping tablet replica
I20251026 21:07:14.779824 11907 raft_consensus.cc:2243] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:14.779986 11906 tablet_replica.cc:333] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f: stopping tablet replica
I20251026 21:07:14.779989 11907 raft_consensus.cc:2272] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:14.780288 11906 raft_consensus.cc:2243] T 7ad4ae3222994d04a6215f3753e81b0d P 643943dcd2704d72852a37536e03a82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:14.780534 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 11383
I20251026 21:07:14.781164 11907 ts_tablet_manager.cc:1916] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:14.783797 11905 tablet_replica.cc:333] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: stopping tablet replica
I20251026 21:07:14.784020 11905 raft_consensus.cc:2243] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:07:14.784245 11907 ts_tablet_manager.cc:1929] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:14.784323 11907 log.cc:1199] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/7ad4ae3222994d04a6215f3753e81b0d
I20251026 21:07:14.784595 11905 raft_consensus.cc:2272] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:14.784633 11907 ts_tablet_manager.cc:1950] T 7ad4ae3222994d04a6215f3753e81b0d P 8b0950373eb54a7886e1314b21421055: Deleting consensus metadata
I20251026 21:07:14.785398 11118 catalog_manager.cc:4985] TS 8b0950373eb54a7886e1314b21421055 (127.3.175.3:42287): tablet 7ad4ae3222994d04a6215f3753e81b0d (table test-table [id=85e5a355b457490faafaa745a4a87ca5]) successfully deleted
I20251026 21:07:14.785605 11905 ts_tablet_manager.cc:1916] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:14.787464 11905 ts_tablet_manager.cc:1929] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:14.787537 11905 log.cc:1199] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/7ad4ae3222994d04a6215f3753e81b0d
I20251026 21:07:14.787792 11905 ts_tablet_manager.cc:1950] T 7ad4ae3222994d04a6215f3753e81b0d P 226478220f964d02baead480f14a9c72: Deleting consensus metadata
I20251026 21:07:14.788471 11118 catalog_manager.cc:4985] TS 226478220f964d02baead480f14a9c72 (127.3.175.2:41823): tablet 7ad4ae3222994d04a6215f3753e81b0d (table test-table [id=85e5a355b457490faafaa745a4a87ca5]) successfully deleted
W20251026 21:07:14.789443 11880 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:14.788924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:40987 (local address 127.0.0.1:42636)
1026 21:07:14.789300 (+ 376us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:14.789352 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:40987: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":233}
I20251026 21:07:14.789647 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 11519
W20251026 21:07:14.789672 11117 catalog_manager.cc:4712] TS 643943dcd2704d72852a37536e03a82f (127.3.175.1:40987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ad4ae3222994d04a6215f3753e81b0d: Network error: Client connection negotiation failed: client connection to 127.3.175.1:40987: connect: Connection reset by peer (error 104)
I20251026 21:07:14.796237 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 11655
I20251026 21:07:14.803767 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 10993
I20251026 21:07:14.810282 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 11105
2025-10-26T21:07:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10736 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:14 dist-test-slave-th6x krb5kdc[11910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:14 dist-test-slave-th6x krb5kdc[11910](info): set up 2 sockets
Oct 26 21:07:14 dist-test-slave-th6x krb5kdc[11910](info): commencing operation
krb5kdc: starting...
W20251026 21:07:16.820140 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:16 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:16.962033 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34987
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:43669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:17.073781 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:17.074038 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:17.074111 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:17.077484 11926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:17.077564 11926 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 21:07:17.077584 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:17.077606 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:17.077625 11926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:17.077649 11926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:17.082113 11926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:34987
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:17.083315 11926 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:17.084254 11926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:17.090267 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.090307 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.090474 11926 server_base.cc:1047] running on GCE node
W20251026 21:07:17.090299 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.091035 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:17.092051 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:17.093219 11926 hybrid_clock.cc:648] HybridClock initialized: now 1761512837093191 us; error 46 us; skew 500 ppm
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:17.095877 11926 init.cc:377] Logged in from keytab as oryx/127.3.175.62@KRBTEST.COM (short username oryx)
I20251026 21:07:17.097026 11926 webserver.cc:492] Webserver started at http://127.3.175.62:44179/ using document root <none> and password file <none>
I20251026 21:07:17.097600 11926 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:17.097658 11926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:17.097847 11926 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:17.099675 11926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "6dce224b11414773a482e966a0a9f227"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "2ae3846e04f6dcaa71db4838b44af66a"
server_key_iv: "25063c56925fbfba5c3f1fda3e49edb7"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.100174 11926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "6dce224b11414773a482e966a0a9f227"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "2ae3846e04f6dcaa71db4838b44af66a"
server_key_iv: "25063c56925fbfba5c3f1fda3e49edb7"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.103766 11926 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:07:17.105942 11941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.106951 11926 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:17.107066 11926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "6dce224b11414773a482e966a0a9f227"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "2ae3846e04f6dcaa71db4838b44af66a"
server_key_iv: "25063c56925fbfba5c3f1fda3e49edb7"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.107164 11926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:17.138675 11926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:17.145361 11926 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:17.145570 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:17.154018 11926 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:34987
I20251026 21:07:17.154253 12003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:34987 every 8 connection(s)
I20251026 21:07:17.155146 11926 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:17.158083 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:17.158794 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 11926
I20251026 21:07:17.158911 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:17.159149 3772 external_mini_cluster.cc:1468] Setting key 00c9ae442edcf6805bf162129e60dc40
I20251026 21:07:17.163748 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Bootstrap starting.
I20251026 21:07:17.166067 12004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Neither blocks nor log segments found. Creating new log.
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:17.166788 12004 log.cc:826] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:17.169165 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: No bootstrap required, opened a new log
I20251026 21:07:17.172114 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.160433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.3.175.62:34987)
1026 21:07:17.160903 (+ 470us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.160914 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.160949 (+ 35us) server_negotiation.cc:408] Connection header received
1026 21:07:17.161473 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.161489 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.161788 (+ 299us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.162114 (+ 326us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.163058 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.163880 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.164560 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.164857 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.167052 (+ 2195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.167089 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.167104 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.167145 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.169399 (+ 2254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.169813 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.169819 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.169825 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.169921 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.170116 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.170119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.170121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.170544 (+ 423us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.170749 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.170929 (+ 180us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.171192 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20251026 21:07:17.173056 12004 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } }
I20251026 21:07:17.173274 12004 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:17.173349 12004 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dce224b11414773a482e966a0a9f227, State: Initialized, Role: FOLLOWER
I20251026 21:07:17.173763 12004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } }
I20251026 21:07:17.173890 12004 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:17.173949 12004 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:17.174042 12004 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:17.174938 12004 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } }
I20251026 21:07:17.175249 12004 leader_election.cc:304] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dce224b11414773a482e966a0a9f227; no voters:
I20251026 21:07:17.175546 12004 leader_election.cc:290] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:17.175663 12009 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:17.175869 12009 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [term 1 LEADER]: Becoming Leader. State: Replica: 6dce224b11414773a482e966a0a9f227, State: Running, Role: LEADER
I20251026 21:07:17.176165 12009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } }
I20251026 21:07:17.176581 12004 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:17.177644 12011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dce224b11414773a482e966a0a9f227. Latest consensus state: current_term: 1 leader_uuid: "6dce224b11414773a482e966a0a9f227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } } }
I20251026 21:07:17.178241 12011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:17.178131 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dce224b11414773a482e966a0a9f227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dce224b11414773a482e966a0a9f227" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 34987 } } }
I20251026 21:07:17.178586 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:17.179051 12023 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:17.181985 12023 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:17.187361 12023 catalog_manager.cc:1357] Generated new cluster ID: dd6bd7e801ff40aeb55e647ac5401c96
I20251026 21:07:17.187439 12023 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:17.203105 12023 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:17.204133 12023 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:17.222532 12023 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Generated new TSK 0
I20251026 21:07:17.223397 12023 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:17.278226 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34987
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:43669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:17.388752 12032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:17.389001 12032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:17.389066 12032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:17.392334 12032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:17.392412 12032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:17.392484 12032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:07:17.392567 12032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:17.396955 12032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:17.398113 12032 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:17.399001 12032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:17.405774 12038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.405794 12037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.405774 12040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.406260 12032 server_base.cc:1047] running on GCE node
I20251026 21:07:17.406615 12032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:17.407133 12032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:17.408282 12032 hybrid_clock.cc:648] HybridClock initialized: now 1761512837408276 us; error 53 us; skew 500 ppm
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:17.411059 12032 init.cc:377] Logged in from keytab as oryx/127.3.175.1@KRBTEST.COM (short username oryx)
I20251026 21:07:17.412256 12032 webserver.cc:492] Webserver started at http://127.3.175.1:33653/ using document root <none> and password file <none>
I20251026 21:07:17.412819 12032 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:17.412878 12032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:17.413075 12032 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:17.414901 12032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "488f7e81f00143059205a1776d98be29"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "f92c8dd77e085de8c7ed215ab2b805c2"
server_key_iv: "dfd43c3e22fbc669366bb2b21924e054"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.415455 12032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "488f7e81f00143059205a1776d98be29"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "f92c8dd77e085de8c7ed215ab2b805c2"
server_key_iv: "dfd43c3e22fbc669366bb2b21924e054"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.418974 12032 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:07:17.421360 12047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.422497 12032 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:17.422626 12032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "488f7e81f00143059205a1776d98be29"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "f92c8dd77e085de8c7ed215ab2b805c2"
server_key_iv: "dfd43c3e22fbc669366bb2b21924e054"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.422735 12032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:17.437182 12032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:17.440304 12032 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:17.440475 12032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:17.441948 12032 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:17.442006 12032 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.442076 12032 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:17.442108 12032 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:17.454128 12032 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:44455
I20251026 21:07:17.455111 12032 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:17.455420 12162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:44455 every 8 connection(s)
I20251026 21:07:17.455581 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.442545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:35343 (local address 127.3.175.62:34987)
1026 21:07:17.442675 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.442678 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.443536 (+ 858us) server_negotiation.cc:408] Connection header received
1026 21:07:17.444456 (+ 920us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.444460 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.444537 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.444634 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.446638 (+ 2004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.447343 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.448347 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.448584 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.451446 (+ 2862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.451478 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.451481 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.451519 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.453449 (+ 1930us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.454082 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.454086 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.454088 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.454168 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.454469 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.454472 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.454474 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.454643 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.454739 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.455242 (+ 503us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.455409 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:07:17.456279 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.442906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.1:35343)
1026 21:07:17.443404 (+ 498us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.443439 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.444250 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.444864 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.444878 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.445521 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.446410 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.446429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.447488 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.447492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.448192 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.448204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.448489 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.449305 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.449360 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.451229 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.453583 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.453592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.453605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.453937 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.454286 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.454289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.454291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.454378 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.454749 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.454757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.454998 (+ 241us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.455196 (+ 198us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.455515 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20251026 21:07:17.457671 12163 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34987
I20251026 21:07:17.457819 12163 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:17.458309 12163 heartbeater.cc:507] Master 127.3.175.62:34987 requested a full tablet report, sending...
I20251026 21:07:17.459710 11957 ts_manager.cc:194] Registered new tserver with Master: 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455)
I20251026 21:07:17.460866 11957 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:35343
I20251026 21:07:17.464470 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12032
I20251026 21:07:17.464570 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:17.464815 3772 external_mini_cluster.cc:1468] Setting key d306a7fd542277c2edc70b7098922fe8
I20251026 21:07:17.466152 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.459786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:43237 (local address 127.3.175.62:34987)
1026 21:07:17.459929 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.459935 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.460205 (+ 270us) server_negotiation.cc:408] Connection header received
1026 21:07:17.460322 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.460325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.460372 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.460444 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.460984 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.461480 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.462140 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.462377 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.463241 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.463258 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.463261 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.463312 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.464642 (+ 1330us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.465052 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.465055 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.465057 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.465103 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.465373 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.465376 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.465378 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.465559 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.465646 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.465924 (+ 278us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.466023 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 21:07:17.466203 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.459672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.1:43237)
1026 21:07:17.460092 (+ 420us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.460109 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.460228 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.460471 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.460474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.460655 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.460873 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.460879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.461589 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.461592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.462032 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.462040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.462149 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.462824 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.462837 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.463125 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.464744 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.464748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.464750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.464949 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.465208 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.465213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.465215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.465272 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.465657 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.465661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.465755 (+ 94us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.465870 (+ 115us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.466005 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:17.514007 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34987
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:43669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:17.625385 12174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:17.625638 12174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:17.625705 12174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:17.629107 12174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:17.629194 12174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:17.629256 12174 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:07:17.629310 12174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:17.635991 12174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:17.637533 12174 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:17.638707 12174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:17.646692 12180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.646740 12182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.646893 12174 server_base.cc:1047] running on GCE node
W20251026 21:07:17.646593 12179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.647382 12174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:17.647994 12174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:17.649103 12174 hybrid_clock.cc:648] HybridClock initialized: now 1761512837649083 us; error 36 us; skew 500 ppm
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:17.651872 12174 init.cc:377] Logged in from keytab as oryx/127.3.175.2@KRBTEST.COM (short username oryx)
I20251026 21:07:17.652938 12174 webserver.cc:492] Webserver started at http://127.3.175.2:33543/ using document root <none> and password file <none>
I20251026 21:07:17.653522 12174 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:17.653573 12174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:17.653755 12174 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:17.655546 12174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "057b9ef5641d9df62cd84b971720b83f"
server_key_iv: "150be5490c1434be66e640008184bda5"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.655997 12174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "057b9ef5641d9df62cd84b971720b83f"
server_key_iv: "150be5490c1434be66e640008184bda5"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.659471 12174 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:07:17.661742 12189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.662784 12174 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:17.662907 12174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "057b9ef5641d9df62cd84b971720b83f"
server_key_iv: "150be5490c1434be66e640008184bda5"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.663003 12174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:17.683120 12174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:17.686375 12174 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:17.686578 12174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:17.688154 12174 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:17.688215 12174 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.688278 12174 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:17.688314 12174 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:17.699637 12174 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:45979
I20251026 21:07:17.699684 12304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:45979 every 8 connection(s)
I20251026 21:07:17.700812 12174 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:17.701153 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.688828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:45127 (local address 127.3.175.62:34987)
1026 21:07:17.688992 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.688997 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.689671 (+ 674us) server_negotiation.cc:408] Connection header received
1026 21:07:17.690613 (+ 942us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.690617 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.690678 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.690758 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.692437 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.693439 (+ 1002us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.694349 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.694588 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.697182 (+ 2594us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.697210 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.697214 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.697251 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.698880 (+ 1629us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.699525 (+ 645us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.699529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.699531 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.699592 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.700037 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.700041 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.700043 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.700215 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.700294 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.700894 (+ 600us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.701022 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 21:07:17.701956 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.689093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.2:45127)
1026 21:07:17.689536 (+ 443us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.689580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.690415 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.690939 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.690947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.691549 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.692268 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.692282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.693582 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.693588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.694223 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.694232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.694467 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.695261 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.695318 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.696997 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.699005 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.699012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.699025 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.699418 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.699704 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.699708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.699710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.699894 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.700321 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.700327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.700608 (+ 281us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.700858 (+ 250us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.701150 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20251026 21:07:17.703187 12305 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34987
I20251026 21:07:17.703377 12305 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:17.703869 12305 heartbeater.cc:507] Master 127.3.175.62:34987 requested a full tablet report, sending...
I20251026 21:07:17.704851 11958 ts_manager.cc:194] Registered new tserver with Master: b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979)
I20251026 21:07:17.705615 11958 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:45127
I20251026 21:07:17.710295 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12174
I20251026 21:07:17.710407 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:17.710650 3772 external_mini_cluster.cc:1468] Setting key 2f51b4df4e37b7dc06f261bd3d0a9215
I20251026 21:07:17.711228 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.705235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:45147 (local address 127.3.175.62:34987)
1026 21:07:17.705361 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.705364 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.705475 (+ 111us) server_negotiation.cc:408] Connection header received
1026 21:07:17.705591 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.705594 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.705631 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.705707 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.706346 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.707029 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.707622 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.707786 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.708523 (+ 737us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.708536 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.708539 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.708555 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.709791 (+ 1236us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.710173 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.710177 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.710178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.710220 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.710432 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.710436 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.710441 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.710611 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.710704 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.710981 (+ 277us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.711087 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:07:17.711241 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.705133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.2:45147)
1026 21:07:17.705396 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.705411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.705516 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.705735 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.705738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.705954 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.706243 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.706251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.707125 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.707128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.707524 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.707532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.707619 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.708150 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.708163 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.708429 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.709890 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.709893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.709895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.710049 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.710305 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.710309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.710310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.710355 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.710741 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.710744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.710814 (+ 70us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.710933 (+ 119us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.711063 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab.
I20251026 21:07:17.759092 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:34987
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:43669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:17.869606 12316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:17.869879 12316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:17.869980 12316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:17.873312 12316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:17.873394 12316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:17.873489 12316 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 21:07:17.873591 12316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:17.878108 12316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:43669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/krb5kdc/oryx.keytab
--principal=oryx/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.175.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:17.879334 12316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:17.880245 12316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:17.887081 12322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.887082 12321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:17.887082 12324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:17.887367 12316 server_base.cc:1047] running on GCE node
I20251026 21:07:17.887816 12316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:17.888413 12316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:17.889568 12316 hybrid_clock.cc:648] HybridClock initialized: now 1761512837889559 us; error 32 us; skew 500 ppm
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:17.892360 12316 init.cc:377] Logged in from keytab as oryx/127.3.175.3@KRBTEST.COM (short username oryx)
I20251026 21:07:17.893397 12316 webserver.cc:492] Webserver started at http://127.3.175.3:40709/ using document root <none> and password file <none>
I20251026 21:07:17.893939 12316 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:17.893983 12316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:17.894140 12316 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:17.895931 12316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "bff96662759e472197a53ac4b20f7df0"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "c6214535b46784173330980c50a06f67"
server_key_iv: "0b80f16afd843eb2922f9ce45d003c41"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.896402 12316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "bff96662759e472197a53ac4b20f7df0"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "c6214535b46784173330980c50a06f67"
server_key_iv: "0b80f16afd843eb2922f9ce45d003c41"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.899796 12316 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:07:17.902137 12331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.903164 12316 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:17.903263 12316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "bff96662759e472197a53ac4b20f7df0"
format_stamp: "Formatted at 2025-10-26 21:07:17 on dist-test-slave-th6x"
server_key: "c6214535b46784173330980c50a06f67"
server_key_iv: "0b80f16afd843eb2922f9ce45d003c41"
server_key_version: "encryptionkey@0"
I20251026 21:07:17.903393 12316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:17.914595 12316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:17.917366 12316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:17.917520 12316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:17.918939 12316 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:17.918995 12316 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:17.919059 12316 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:17.919095 12316 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:17.931006 12316 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:44523
I20251026 21:07:17.931046 12446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:44523 every 8 connection(s)
I20251026 21:07:17.932056 12316 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:17.932262 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.919568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:36663 (local address 127.3.175.62:34987)
1026 21:07:17.919724 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.919729 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.920585 (+ 856us) server_negotiation.cc:408] Connection header received
1026 21:07:17.921759 (+ 1174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.921762 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.921813 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.921893 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.923784 (+ 1891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.924478 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.925185 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.925365 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.928215 (+ 2850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.928236 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.928239 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.928269 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.930099 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.930725 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.930729 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.930731 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.930809 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.931132 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.931138 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.931142 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.931357 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.931453 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.931956 (+ 503us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.932084 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251026 21:07:17.932907 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.919862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.3:36663)
1026 21:07:17.920411 (+ 549us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.920457 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.921537 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.922098 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.922111 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.922740 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.923598 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.923618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.924601 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.924605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.925063 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.925072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.925296 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.925944 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.925968 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.928021 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.930221 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.930230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.930245 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.930580 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.930926 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.930930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.930932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.931019 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.931475 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.931482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.931714 (+ 232us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.931911 (+ 197us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.932187 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20251026 21:07:17.934207 12447 heartbeater.cc:344] Connected to a master server at 127.3.175.62:34987
I20251026 21:07:17.934370 12447 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:17.934981 12447 heartbeater.cc:507] Master 127.3.175.62:34987 requested a full tablet report, sending...
I20251026 21:07:17.935257 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12316
I20251026 21:07:17.935411 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:17.935681 3772 external_mini_cluster.cc:1468] Setting key ec0b6f1f9e4dae3d191ab2267a8a454d
I20251026 21:07:17.936028 11957 ts_manager.cc:194] Registered new tserver with Master: bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523)
I20251026 21:07:17.936605 11957 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:36663
I20251026 21:07:17.937674 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:07:17.944450 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.937422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:56051 (local address 127.3.175.62:34987)
1026 21:07:17.937546 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.937551 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.937564 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:17.937595 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.937598 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.937644 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.937727 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.938409 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.939157 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.939832 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.940050 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.940777 (+ 727us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.940799 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.940806 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.940838 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.942580 (+ 1742us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.943156 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.943159 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.943161 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.943205 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.943527 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.943530 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.943531 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.943684 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.943762 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.944094 (+ 332us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.944225 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 21:07:17.944463 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.937058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.3.175.3:56051)
1026 21:07:17.937344 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.937361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.937455 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.937763 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.937766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.937986 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:17.938275 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.938284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.939273 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.939275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.939711 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.939719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.939815 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.940367 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:17.940380 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:17.940663 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:17.942741 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.942747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.942751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.943044 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.943333 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:17.943340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:17.943346 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:17.943434 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:17.943775 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:17.943782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:17.943901 (+ 119us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.944050 (+ 149us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.944224 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":127,"threads_started":1}
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
W20251026 21:07:17.961568 12007 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:17.955687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.3.175.62:34987)
1026 21:07:17.955845 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.955849 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.955861 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:17.955898 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.955901 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.955943 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.956016 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.957048 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.957536 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.958209 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.958451 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.961434 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52158: BlockingRecv error: recv got EOF from 127.0.0.1:52158 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 26 21:07:17 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:17.970217 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.962978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.3.175.62:34987)
1026 21:07:17.963104 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.963112 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.963142 (+ 30us) server_negotiation.cc:408] Connection header received
1026 21:07:17.963309 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.963314 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.963363 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.963432 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:17.964185 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.964659 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.965259 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.965474 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.967678 (+ 2204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:17.967696 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:17.967698 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:17.967724 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:17.968993 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.969381 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.969384 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.969386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.969429 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:17.969588 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:17.969590 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:17.969591 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:17.969729 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:17.969830 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.969999 (+ 169us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.970087 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:07:17.972784 11957 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:07:17.975198 11957 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:17.988140 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.983618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:45979 (local address 127.0.0.1:52462)
1026 21:07:17.984277 (+ 659us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.984322 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.984483 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.985284 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.985291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.985314 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:17.985600 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.985607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.986705 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.986711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.987582 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.987591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.987693 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.987753 (+ 60us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.987848 (+ 95us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.987968 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":103,"threads_started":1}
I20251026 21:07:17.989041 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.984347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:44523 (local address 127.0.0.1:49212)
1026 21:07:17.985033 (+ 686us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.985046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.985136 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.985954 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.985957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.985978 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:17.986183 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.986190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.987365 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.987369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.988659 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.988667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.988770 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.988793 (+ 23us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.988843 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.988918 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":94,"threads_started":1}
I20251026 21:07:17.989064 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.984392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.3.175.1:44455)
1026 21:07:17.984904 (+ 512us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.984911 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.984931 (+ 20us) server_negotiation.cc:408] Connection header received
1026 21:07:17.985014 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.985021 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.985186 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.985324 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:17.985788 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.986815 (+ 1027us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.988070 (+ 1255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.988641 (+ 571us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.988764 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.988833 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.988930 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":67,"threads_started":1}
I20251026 21:07:17.989229 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.983786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.3.175.2:45979)
1026 21:07:17.984217 (+ 431us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.984224 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.984626 (+ 402us) server_negotiation.cc:408] Connection header received
1026 21:07:17.984719 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.984725 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.984930 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.985077 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:17.985722 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.986603 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.988368 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.988847 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.988947 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.989007 (+ 60us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.989096 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20251026 21:07:17.989447 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.983850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.0.0.1:36560)
1026 21:07:17.984643 (+ 793us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:17.984658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:17.984745 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:17.985316 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:17.985322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:17.985343 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:17.985645 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.985654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.986943 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.986946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:17.987778 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:17.987785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.987878 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.987892 (+ 14us) client_negotiation.cc:769] Sending connection context
1026 21:07:17.987938 (+ 46us) client_negotiation.cc:240] Negotiation successful
1026 21:07:17.987978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":95,"threads_started":1}
I20251026 21:07:17.990047 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:17.984510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.3.175.3:44523)
1026 21:07:17.985548 (+ 1038us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:17.985555 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:17.985575 (+ 20us) server_negotiation.cc:408] Connection header received
1026 21:07:17.985650 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:17.985655 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:17.985819 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:17.985961 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:17.986307 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.987237 (+ 930us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:17.988795 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:17.989664 (+ 869us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:17.989760 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:17.989820 (+ 60us) server_negotiation.cc:300] Negotiation successful
1026 21:07:17.989907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":118,"threads_started":1}
I20251026 21:07:17.992233 12097 tablet_service.cc:1505] Processing CreateTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 (DEFAULT_TABLE table=test-table [id=58de548fa28c451fb89252d458880468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:17.992360 12381 tablet_service.cc:1505] Processing CreateTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 (DEFAULT_TABLE table=test-table [id=58de548fa28c451fb89252d458880468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:17.992267 12239 tablet_service.cc:1505] Processing CreateTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 (DEFAULT_TABLE table=test-table [id=58de548fa28c451fb89252d458880468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:17.993086 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17970fe7a1a4d64a8161885c23ffab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:17.993117 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17970fe7a1a4d64a8161885c23ffab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:17.993126 12381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17970fe7a1a4d64a8161885c23ffab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:17.998481 12475 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Bootstrap starting.
I20251026 21:07:18.001036 12475 tablet_bootstrap.cc:654] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.001756 12475 log.cc:826] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:18.002403 12476 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Bootstrap starting.
I20251026 21:07:18.002811 12477 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Bootstrap starting.
I20251026 21:07:18.003790 12475 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: No bootstrap required, opened a new log
I20251026 21:07:18.004050 12475 ts_tablet_manager.cc:1403] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 21:07:18.004766 12477 tablet_bootstrap.cc:654] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.004807 12476 tablet_bootstrap.cc:654] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.005996 12477 log.cc:826] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:18.005997 12476 log.cc:826] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:18.007544 12475 raft_consensus.cc:359] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.007771 12475 raft_consensus.cc:385] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.007822 12477 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: No bootstrap required, opened a new log
I20251026 21:07:18.007822 12476 tablet_bootstrap.cc:492] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: No bootstrap required, opened a new log
I20251026 21:07:18.007839 12475 raft_consensus.cc:740] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488f7e81f00143059205a1776d98be29, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.007999 12477 ts_tablet_manager.cc:1403] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:07:18.008000 12476 ts_tablet_manager.cc:1403] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 21:07:18.008384 12475 consensus_queue.cc:260] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.009238 12475 ts_tablet_manager.cc:1434] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:07:18.010484 12477 raft_consensus.cc:359] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.010537 12476 raft_consensus.cc:359] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.010664 12477 raft_consensus.cc:385] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.010681 12476 raft_consensus.cc:385] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.010735 12476 raft_consensus.cc:740] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc42a6d0554610b48ec1d0eae343f6, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.010738 12477 raft_consensus.cc:740] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.010957 12163 heartbeater.cc:499] Master 127.3.175.62:34987 was elected leader, sending a full tablet report...
I20251026 21:07:18.011219 12476 consensus_queue.cc:260] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.011212 12477 consensus_queue.cc:260] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.011924 12447 heartbeater.cc:499] Master 127.3.175.62:34987 was elected leader, sending a full tablet report...
I20251026 21:07:18.012094 12477 ts_tablet_manager.cc:1434] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251026 21:07:18.012327 12305 heartbeater.cc:499] Master 127.3.175.62:34987 was elected leader, sending a full tablet report...
I20251026 21:07:18.013226 12476 ts_tablet_manager.cc:1434] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251026 21:07:18.026551 12483 raft_consensus.cc:493] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:18.026739 12483 raft_consensus.cc:515] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.027918 12483 leader_election.cc:290] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455)
I20251026 21:07:18.031565 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.028158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.3.175.3:37699)
1026 21:07:18.028391 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.028409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.028527 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.028797 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.028799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.028812 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:18.029089 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.029098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.030097 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.030100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.031085 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.031097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.031225 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.031250 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.031347 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.031409 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20251026 21:07:18.031579 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.028158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:45979 (local address 127.3.175.3:44623)
1026 21:07:18.028602 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.028617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.028692 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.028938 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.028941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.028955 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:18.029216 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.029223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.030165 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.030168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.031095 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.031101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.031225 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.031250 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.031341 (+ 91us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.031409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":50,"thread_start_us":138,"threads_started":1}
I20251026 21:07:18.032038 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.028254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:44623 (local address 127.3.175.2:45979)
1026 21:07:18.028413 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.028418 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.028691 (+ 273us) server_negotiation.cc:408] Connection header received
1026 21:07:18.028782 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.028785 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.028832 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.028915 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:18.029323 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.030058 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.031210 (+ 1152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.031738 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.031787 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.031841 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.031917 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:07:18.032147 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.028254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:37699 (local address 127.3.175.1:44455)
1026 21:07:18.028391 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.028396 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.028494 (+ 98us) server_negotiation.cc:408] Connection header received
1026 21:07:18.028626 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.028629 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.028687 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.028779 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:18.029219 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.029992 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.031210 (+ 1218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.031889 (+ 679us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.031922 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.031969 (+ 47us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.032021 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 21:07:18.032809 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a17970fe7a1a4d64a8161885c23ffab7" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" is_pre_election: true
I20251026 21:07:18.033083 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a17970fe7a1a4d64a8161885c23ffab7" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29" is_pre_election: true
I20251026 21:07:18.033135 12259 raft_consensus.cc:2468] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.033329 12117 raft_consensus.cc:2468] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.033617 12335 leader_election.cc:304] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7fc42a6d0554610b48ec1d0eae343f6, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.033898 12483 raft_consensus.cc:2804] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:18.033969 12483 raft_consensus.cc:493] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:18.034004 12483 raft_consensus.cc:3060] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.035054 12483 raft_consensus.cc:515] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.035498 12483 leader_election.cc:290] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Requested vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455)
I20251026 21:07:18.035872 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a17970fe7a1a4d64a8161885c23ffab7" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29"
I20251026 21:07:18.035887 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a17970fe7a1a4d64a8161885c23ffab7" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
I20251026 21:07:18.036007 12117 raft_consensus.cc:3060] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.036007 12259 raft_consensus.cc:3060] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.037226 12117 raft_consensus.cc:2468] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.037226 12259 raft_consensus.cc:2468] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.037609 12335 leader_election.cc:304] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7fc42a6d0554610b48ec1d0eae343f6, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.037844 12483 raft_consensus.cc:2804] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:18.038072 12483 raft_consensus.cc:697] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Becoming Leader. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Running, Role: LEADER
I20251026 21:07:18.038383 12483 consensus_queue.cc:237] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.041505 11958 catalog_manager.cc:5649] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bff96662759e472197a53ac4b20f7df0 (127.3.175.3). New cstate: current_term: 1 leader_uuid: "bff96662759e472197a53ac4b20f7df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } health_report { overall_health: HEALTHY } } }
I20251026 21:07:18.053221 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.050202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.3.175.3:44523)
1026 21:07:18.050344 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.050350 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.050365 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:18.050445 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.050447 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.050499 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.050587 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.050925 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.051484 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.052119 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.052308 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.052376 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.052829 (+ 453us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.052940 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.053021 (+ 81us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.053075 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 21:07:18.058642 12117 raft_consensus.cc:1275] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.058662 12259 raft_consensus.cc:1275] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.059476 12485 consensus_queue.cc:1048] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.059677 12483 consensus_queue.cc:1048] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.066067 12488 mvcc.cc:204] Tried to move back new op lower bound from 7215156584681127936 to 7215156584607952896. Current Snapshot: MvccSnapshot[applied={T|T < 7215156584681127936}]
I20251026 21:07:18.074579 11957 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.074733 11957 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.077097 11957 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Sending DeleteTablet for 3 replicas of tablet a17970fe7a1a4d64a8161885c23ffab7
I20251026 21:07:18.077936 12097 tablet_service.cc:1552] Processing DeleteTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:36560
I20251026 21:07:18.078151 12239 tablet_service.cc:1552] Processing DeleteTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52462
I20251026 21:07:18.078151 12381 tablet_service.cc:1552] Processing DeleteTablet for tablet a17970fe7a1a4d64a8161885c23ffab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:49212
I20251026 21:07:18.078712 12498 tablet_replica.cc:333] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: stopping tablet replica
I20251026 21:07:18.079007 12498 raft_consensus.cc:2243] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.079083 12497 tablet_replica.cc:333] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: stopping tablet replica
I20251026 21:07:18.079372 12497 raft_consensus.cc:2243] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.079370 11957 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 21:07:18.079713 12497 raft_consensus.cc:2272] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 21:07:18.079823 11957 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:18.080758 12498 raft_consensus.cc:2272] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.081280 12497 ts_tablet_manager.cc:1916] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.082269 12498 ts_tablet_manager.cc:1916] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.084242 12497 ts_tablet_manager.cc:1929] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.084415 12497 log.cc:1199] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/a17970fe7a1a4d64a8161885c23ffab7
I20251026 21:07:18.084841 12497 ts_tablet_manager.cc:1950] T a17970fe7a1a4d64a8161885c23ffab7 P 488f7e81f00143059205a1776d98be29: Deleting consensus metadata
I20251026 21:07:18.084910 12498 ts_tablet_manager.cc:1929] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.085005 12498 log.cc:1199] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/a17970fe7a1a4d64a8161885c23ffab7
I20251026 21:07:18.085368 12498 ts_tablet_manager.cc:1950] T a17970fe7a1a4d64a8161885c23ffab7 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting consensus metadata
I20251026 21:07:18.086242 12499 tablet_replica.cc:333] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: stopping tablet replica
I20251026 21:07:18.086395 11942 catalog_manager.cc:4985] TS 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455): tablet a17970fe7a1a4d64a8161885c23ffab7 (table test-table [id=58de548fa28c451fb89252d458880468]) successfully deleted
I20251026 21:07:18.086526 12499 raft_consensus.cc:2243] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:07:18.086912 12499 raft_consensus.cc:2272] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.087419 12239 tablet_service.cc:1505] Processing CreateTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a (DEFAULT_TABLE table=test-table [id=05f4cc6da4ff402192636aa63e647ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.087670 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cffc8736ba411cbd3c48400eabeb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.089076 12499 ts_tablet_manager.cc:1916] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.089895 12476 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Bootstrap starting.
I20251026 21:07:18.090262 12097 tablet_service.cc:1505] Processing CreateTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a (DEFAULT_TABLE table=test-table [id=05f4cc6da4ff402192636aa63e647ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.090507 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cffc8736ba411cbd3c48400eabeb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.090914 12476 tablet_bootstrap.cc:654] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.091617 11945 catalog_manager.cc:4985] TS b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979): tablet a17970fe7a1a4d64a8161885c23ffab7 (table test-table [id=58de548fa28c451fb89252d458880468]) successfully deleted
I20251026 21:07:18.091773 12499 ts_tablet_manager.cc:1929] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.091848 12499 log.cc:1199] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/a17970fe7a1a4d64a8161885c23ffab7
I20251026 21:07:18.092224 12499 ts_tablet_manager.cc:1950] T a17970fe7a1a4d64a8161885c23ffab7 P bff96662759e472197a53ac4b20f7df0: Deleting consensus metadata
I20251026 21:07:18.092273 12381 tablet_service.cc:1505] Processing CreateTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a (DEFAULT_TABLE table=test-table [id=05f4cc6da4ff402192636aa63e647ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.092527 12381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cffc8736ba411cbd3c48400eabeb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.093369 11945 catalog_manager.cc:4985] TS bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523): tablet a17970fe7a1a4d64a8161885c23ffab7 (table test-table [id=58de548fa28c451fb89252d458880468]) successfully deleted
I20251026 21:07:18.094336 12477 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Bootstrap starting.
I20251026 21:07:18.095368 12477 tablet_bootstrap.cc:654] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.096004 12475 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Bootstrap starting.
I20251026 21:07:18.096482 12477 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: No bootstrap required, opened a new log
I20251026 21:07:18.096554 12477 ts_tablet_manager.cc:1403] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 21:07:18.096899 12477 raft_consensus.cc:359] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.096992 12477 raft_consensus.cc:385] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.097012 12477 raft_consensus.cc:740] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.097028 12475 tablet_bootstrap.cc:654] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.097121 12477 consensus_queue.cc:260] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.097378 12477 ts_tablet_manager.cc:1434] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 21:07:18.098567 12475 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: No bootstrap required, opened a new log
I20251026 21:07:18.098659 12475 ts_tablet_manager.cc:1403] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:18.099099 12476 tablet_bootstrap.cc:492] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: No bootstrap required, opened a new log
I20251026 21:07:18.099117 12475 raft_consensus.cc:359] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.099216 12476 ts_tablet_manager.cc:1403] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251026 21:07:18.099227 12475 raft_consensus.cc:385] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.099264 12475 raft_consensus.cc:740] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488f7e81f00143059205a1776d98be29, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.099452 12475 consensus_queue.cc:260] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.099756 12476 raft_consensus.cc:359] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.099828 12475 ts_tablet_manager.cc:1434] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.099865 12476 raft_consensus.cc:385] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.099901 12476 raft_consensus.cc:740] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc42a6d0554610b48ec1d0eae343f6, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.100055 12476 consensus_queue.cc:260] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.100349 12476 ts_tablet_manager.cc:1434] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.160470 12485 raft_consensus.cc:493] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:18.160614 12485 raft_consensus.cc:515] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.161110 12485 leader_election.cc:290] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455), b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979)
I20251026 21:07:18.161617 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36cffc8736ba411cbd3c48400eabeb4a" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29" is_pre_election: true
I20251026 21:07:18.161767 12117 raft_consensus.cc:2468] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.161788 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36cffc8736ba411cbd3c48400eabeb4a" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" is_pre_election: true
I20251026 21:07:18.161947 12259 raft_consensus.cc:2468] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.162140 12332 leader_election.cc:304] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.162372 12485 raft_consensus.cc:2804] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:18.162423 12485 raft_consensus.cc:493] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:18.162492 12485 raft_consensus.cc:3060] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.163736 12485 raft_consensus.cc:515] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.164110 12485 leader_election.cc:290] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Requested vote from peers 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455), b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979)
I20251026 21:07:18.164613 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36cffc8736ba411cbd3c48400eabeb4a" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
I20251026 21:07:18.164656 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36cffc8736ba411cbd3c48400eabeb4a" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29"
I20251026 21:07:18.164736 12259 raft_consensus.cc:3060] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.164758 12117 raft_consensus.cc:3060] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.165831 12117 raft_consensus.cc:2468] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.165831 12259 raft_consensus.cc:2468] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.166167 12335 leader_election.cc:304] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7fc42a6d0554610b48ec1d0eae343f6, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.166445 12485 raft_consensus.cc:2804] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:18.166543 12485 raft_consensus.cc:697] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Becoming Leader. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Running, Role: LEADER
I20251026 21:07:18.166673 12485 consensus_queue.cc:237] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.168459 11956 catalog_manager.cc:5649] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bff96662759e472197a53ac4b20f7df0 (127.3.175.3). New cstate: current_term: 1 leader_uuid: "bff96662759e472197a53ac4b20f7df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } health_report { overall_health: HEALTHY } } }
W20251026 21:07:18.183441 12448 tablet.cc:2378] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:07:18.202636 12306 tablet.cc:2378] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 21:07:18.206598 12164 tablet.cc:2378] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512837, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.62@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:18.217815 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.209147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:34987 (local address 127.0.0.1:52174)
1026 21:07:18.209439 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.209453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.209562 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.209819 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.209821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.210221 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:18.210411 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.210418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.211446 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.211450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.211929 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.211938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.212054 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.212641 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:18.212654 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:18.214283 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:18.216487 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:18.216495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:18.216499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:18.216725 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:18.216973 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:18.216976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:18.216977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:18.217019 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:18.217378 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:18.217383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:18.217478 (+ 95us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.217565 (+ 87us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.217678 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20251026 21:07:18.217952 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.209249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.3.175.62:34987)
1026 21:07:18.209392 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.209397 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.209520 (+ 123us) server_negotiation.cc:408] Connection header received
1026 21:07:18.209647 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.209651 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.209704 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.209804 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:18.210533 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.211346 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.212079 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.212268 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.214405 (+ 2137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:18.214426 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:18.214429 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:18.214456 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:18.216358 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:18.216823 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:18.216826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:18.216828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:18.216881 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:18.217089 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:18.217092 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:18.217094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:18.217287 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:18.217390 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.217633 (+ 243us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.217761 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:07:18.221760 11956 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52174:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 21:07:18.222517 11956 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:18.227550 12097 tablet_service.cc:1505] Processing CreateTablet for tablet db0e21f219b340d4a12d1ba6251f9857 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e8506b0a87e49098a8e87ded527bab9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:07:18.227667 12239 tablet_service.cc:1505] Processing CreateTablet for tablet db0e21f219b340d4a12d1ba6251f9857 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e8506b0a87e49098a8e87ded527bab9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:07:18.227988 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0e21f219b340d4a12d1ba6251f9857. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.227986 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0e21f219b340d4a12d1ba6251f9857. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.230957 12475 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: Bootstrap starting.
I20251026 21:07:18.231812 12381 tablet_service.cc:1505] Processing CreateTablet for tablet db0e21f219b340d4a12d1ba6251f9857 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e8506b0a87e49098a8e87ded527bab9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 21:07:18.232138 12381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0e21f219b340d4a12d1ba6251f9857. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.232187 12475 tablet_bootstrap.cc:654] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.233319 12476 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: Bootstrap starting.
I20251026 21:07:18.234392 12476 tablet_bootstrap.cc:654] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.234437 12477 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: Bootstrap starting.
I20251026 21:07:18.235512 12477 tablet_bootstrap.cc:654] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.235884 12476 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: No bootstrap required, opened a new log
I20251026 21:07:18.235978 12476 ts_tablet_manager.cc:1403] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:18.236543 12476 raft_consensus.cc:359] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.236658 12476 raft_consensus.cc:385] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.236696 12476 raft_consensus.cc:740] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc42a6d0554610b48ec1d0eae343f6, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.236908 12476 consensus_queue.cc:260] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.237118 12475 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: No bootstrap required, opened a new log
I20251026 21:07:18.237223 12475 ts_tablet_manager.cc:1403] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251026 21:07:18.237198 12482 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.237308 12482 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.237756 12475 raft_consensus.cc:359] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.237880 12475 raft_consensus.cc:385] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.237926 12475 raft_consensus.cc:740] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488f7e81f00143059205a1776d98be29, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.238078 12475 consensus_queue.cc:260] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.238184 12477 tablet_bootstrap.cc:492] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: No bootstrap required, opened a new log
I20251026 21:07:18.238276 12477 ts_tablet_manager.cc:1403] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251026 21:07:18.238349 12475 ts_tablet_manager.cc:1434] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.238762 12477 raft_consensus.cc:359] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.238866 12477 raft_consensus.cc:385] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.238899 12477 raft_consensus.cc:740] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.239154 12481 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.239271 12481 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.239220 12477 consensus_queue.cc:260] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.239563 12477 ts_tablet_manager.cc:1434] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.239552 12485 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.239686 12485 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.240792 12476 ts_tablet_manager.cc:1434] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 21:07:18.302446 12481 raft_consensus.cc:493] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:18.302592 12481 raft_consensus.cc:515] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.303743 12481 leader_election.cc:290] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523)
I20251026 21:07:18.307678 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.304585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:44523 (local address 127.3.175.1:34435)
1026 21:07:18.304850 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.304861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.304945 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.305177 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.305180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.305195 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:18.305457 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.305462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.306343 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.306346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.307180 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.307190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.307344 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.307369 (+ 25us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.307425 (+ 56us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.307491 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20251026 21:07:18.308007 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.304715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:34435 (local address 127.3.175.3:44523)
1026 21:07:18.304848 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.304854 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.304914 (+ 60us) server_negotiation.cc:408] Connection header received
1026 21:07:18.305038 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.305040 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.305083 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.305148 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:18.305559 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.306252 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.307335 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.307710 (+ 375us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.307750 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.307810 (+ 60us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.307890 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251026 21:07:18.308244 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.304062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:45979 (local address 127.3.175.1:59451)
1026 21:07:18.304437 (+ 375us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.304453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.304577 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.304937 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.304941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.304954 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:18.305276 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.305287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.306228 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.306230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.307180 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.307190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.307938 (+ 748us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.307958 (+ 20us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.308041 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.308102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":134,"threads_started":1}
I20251026 21:07:18.308302 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.304154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:59451 (local address 127.3.175.2:45979)
1026 21:07:18.304287 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.304291 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.304531 (+ 240us) server_negotiation.cc:408] Connection header received
1026 21:07:18.304750 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.304753 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.304814 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.304915 (+ 101us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:18.305400 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.306107 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.307356 (+ 1249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.307822 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.307866 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.308104 (+ 238us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.308190 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251026 21:07:18.308542 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0e21f219b340d4a12d1ba6251f9857" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff96662759e472197a53ac4b20f7df0" is_pre_election: true
I20251026 21:07:18.308754 12398 raft_consensus.cc:2468] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 0.
I20251026 21:07:18.308808 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0e21f219b340d4a12d1ba6251f9857" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" is_pre_election: true
I20251026 21:07:18.308943 12259 raft_consensus.cc:2468] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 0.
I20251026 21:07:18.309187 12051 leader_election.cc:304] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.309420 12481 raft_consensus.cc:2804] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:18.309484 12481 raft_consensus.cc:493] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:18.309527 12481 raft_consensus.cc:3060] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.310438 12481 raft_consensus.cc:515] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.310778 12481 leader_election.cc:290] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 election: Requested vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523)
I20251026 21:07:18.311200 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0e21f219b340d4a12d1ba6251f9857" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff96662759e472197a53ac4b20f7df0"
I20251026 21:07:18.311216 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0e21f219b340d4a12d1ba6251f9857" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
I20251026 21:07:18.311344 12398 raft_consensus.cc:3060] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.311358 12259 raft_consensus.cc:3060] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.312214 12259 raft_consensus.cc:2468] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 1.
I20251026 21:07:18.312310 12398 raft_consensus.cc:2468] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 1.
I20251026 21:07:18.312492 12051 leader_election.cc:304] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, b7fc42a6d0554610b48ec1d0eae343f6; no voters:
I20251026 21:07:18.312675 12481 raft_consensus.cc:2804] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:18.312894 12481 raft_consensus.cc:697] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [term 1 LEADER]: Becoming Leader. State: Replica: 488f7e81f00143059205a1776d98be29, State: Running, Role: LEADER
I20251026 21:07:18.313131 12481 consensus_queue.cc:237] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.314348 12515 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: TxnStatusTablet state changed. Reason: New leader 488f7e81f00143059205a1776d98be29. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.314458 12515 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:07:18.314728 12517 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 21:07:18.314888 12517 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 21:07:18.315600 11957 catalog_manager.cc:5649] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 488f7e81f00143059205a1776d98be29 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } health_report { overall_health: UNKNOWN } } }
I20251026 21:07:18.345659 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.342273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.0.0.1:36562)
1026 21:07:18.342450 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.342468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.342602 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.342929 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.342932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.342941 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:07:18.343146 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.343152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.343900 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.343903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.344324 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.344332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.344444 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.344498 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.345317 (+ 819us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.345337 (+ 20us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.345430 (+ 93us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.345513 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251026 21:07:18.345659 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.3.175.1:44455)
1026 21:07:18.342523 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.342528 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.342542 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:18.342754 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.342757 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.342819 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.342912 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.343260 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.343772 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.344445 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.344646 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.344702 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.345209 (+ 507us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.345319 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.345457 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.345513 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251026 21:07:18.349133 12259 raft_consensus.cc:1275] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Refusing update from remote peer 488f7e81f00143059205a1776d98be29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.349365 12398 raft_consensus.cc:1275] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Refusing update from remote peer 488f7e81f00143059205a1776d98be29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.349854 12481 consensus_queue.cc:1048] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.350003 12515 consensus_queue.cc:1048] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.351563 12482 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: TxnStatusTablet state changed. Reason: New leader 488f7e81f00143059205a1776d98be29. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.351678 12482 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.352412 12485 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: TxnStatusTablet state changed. Reason: New leader 488f7e81f00143059205a1776d98be29. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.352521 12485 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.353338 12481 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.353441 12481 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:07:18.353534 12490 mvcc.cc:204] Tried to move back new op lower bound from 7215156585872793600 to 7215156585733017600. Current Snapshot: MvccSnapshot[applied={T|T < 7215156585872793600}]
I20251026 21:07:18.355113 12489 mvcc.cc:204] Tried to move back new op lower bound from 7215156585872793600 to 7215156585733017600. Current Snapshot: MvccSnapshot[applied={T|T < 7215156585872793600}]
I20251026 21:07:18.355332 12482 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.355430 12482 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P b7fc42a6d0554610b48ec1d0eae343f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.357973 12515 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.358090 12515 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P 488f7e81f00143059205a1776d98be29: This TxnStatusTablet replica's current role is: LEADER
I20251026 21:07:18.358381 12485 tablet_replica.cc:442] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } } }
I20251026 21:07:18.358480 12485 tablet_replica.cc:445] T db0e21f219b340d4a12d1ba6251f9857 P bff96662759e472197a53ac4b20f7df0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 21:07:18.369311 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.366658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:39805 (local address 127.3.175.1:44455)
1026 21:07:18.366792 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.366798 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.366811 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:18.366960 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.366963 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.367010 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.367077 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.367476 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.367960 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.368617 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.368815 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.368860 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.368942 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.369009 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.369116 (+ 107us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.369165 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 21:07:18.369311 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.366572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.3.175.3:39805)
1026 21:07:18.366724 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.366742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.366842 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.367093 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.367096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.367104 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:07:18.367350 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.367361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.368072 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.368076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.368503 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.368510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.368623 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.368675 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.369031 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.369039 (+ 8us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.369101 (+ 62us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.369189 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 21:07:18.376140 12117 raft_consensus.cc:1275] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.376211 12259 raft_consensus.cc:1275] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.376639 12483 consensus_queue.cc:1048] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.376773 12485 consensus_queue.cc:1048] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.400519 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.397199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:33543 (local address 127.3.175.3:44523)
1026 21:07:18.397325 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.397329 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.397341 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:18.397533 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.397537 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.397599 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.397680 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.398147 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.398906 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.399768 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.399975 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.400023 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.400122 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.400196 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.400320 (+ 124us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.400379 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 21:07:18.400543 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.396911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:44523 (local address 127.3.175.1:33543)
1026 21:07:18.397238 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.397256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.397354 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.397696 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.397699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.397709 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:07:18.397962 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.397968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.399028 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.399031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.399637 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.399648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.399762 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.399816 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.400228 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.400234 (+ 6us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.400301 (+ 67us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.400381 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":111,"threads_started":1}
I20251026 21:07:18.427260 11956 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.427491 11956 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52168:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.429731 11956 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Sending DeleteTablet for 3 replicas of tablet 36cffc8736ba411cbd3c48400eabeb4a
I20251026 21:07:18.430385 12239 tablet_service.cc:1552] Processing DeleteTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52462
I20251026 21:07:18.430470 12381 tablet_service.cc:1552] Processing DeleteTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:49212
I20251026 21:07:18.430619 12498 tablet_replica.cc:333] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: stopping tablet replica
I20251026 21:07:18.430650 12499 tablet_replica.cc:333] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: stopping tablet replica
I20251026 21:07:18.430761 12499 raft_consensus.cc:2243] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Raft consensus shutting down.
I20251026 21:07:18.430758 12498 raft_consensus.cc:2243] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.430794 12097 tablet_service.cc:1552] Processing DeleteTablet for tablet 36cffc8736ba411cbd3c48400eabeb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:36560
I20251026 21:07:18.430904 12498 raft_consensus.cc:2272] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.430974 12499 raft_consensus.cc:2272] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.430998 12497 tablet_replica.cc:333] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: stopping tablet replica
I20251026 21:07:18.431110 12497 raft_consensus.cc:2243] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.431242 12497 raft_consensus.cc:2272] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.432085 12499 ts_tablet_manager.cc:1916] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.432163 12497 ts_tablet_manager.cc:1916] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.435143 12497 ts_tablet_manager.cc:1929] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:07:18.435230 12497 log.cc:1199] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/36cffc8736ba411cbd3c48400eabeb4a
I20251026 21:07:18.435485 12497 ts_tablet_manager.cc:1950] T 36cffc8736ba411cbd3c48400eabeb4a P 488f7e81f00143059205a1776d98be29: Deleting consensus metadata
I20251026 21:07:18.436210 12499 ts_tablet_manager.cc:1929] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:07:18.436291 12499 log.cc:1199] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/36cffc8736ba411cbd3c48400eabeb4a
I20251026 21:07:18.436296 11942 catalog_manager.cc:4985] TS 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455): tablet 36cffc8736ba411cbd3c48400eabeb4a (table test-table [id=05f4cc6da4ff402192636aa63e647ef6]) successfully deleted
I20251026 21:07:18.436537 12499 ts_tablet_manager.cc:1950] T 36cffc8736ba411cbd3c48400eabeb4a P bff96662759e472197a53ac4b20f7df0: Deleting consensus metadata
I20251026 21:07:18.437273 11945 catalog_manager.cc:4985] TS bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523): tablet 36cffc8736ba411cbd3c48400eabeb4a (table test-table [id=05f4cc6da4ff402192636aa63e647ef6]) successfully deleted
I20251026 21:07:18.437371 12498 ts_tablet_manager.cc:1916] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.439590 12498 ts_tablet_manager.cc:1929] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:07:18.439675 12498 log.cc:1199] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/36cffc8736ba411cbd3c48400eabeb4a
I20251026 21:07:18.439939 12498 ts_tablet_manager.cc:1950] T 36cffc8736ba411cbd3c48400eabeb4a P b7fc42a6d0554610b48ec1d0eae343f6: Deleting consensus metadata
I20251026 21:07:18.440600 11945 catalog_manager.cc:4985] TS b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979): tablet 36cffc8736ba411cbd3c48400eabeb4a (table test-table [id=05f4cc6da4ff402192636aa63e647ef6]) successfully deleted
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
W20251026 21:07:18.462013 12007 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:18.454384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.3.175.62:34987)
1026 21:07:18.454525 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.454531 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.454784 (+ 253us) server_negotiation.cc:408] Connection header received
1026 21:07:18.454936 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.454940 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.455004 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.455143 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:18.456243 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.457064 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.458067 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.458291 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.461882 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52188: BlockingRecv error: recv got EOF from 127.0.0.1:52188 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM
I20251026 21:07:18.474367 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.464261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.3.175.62:34987)
1026 21:07:18.464401 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.464407 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.464826 (+ 419us) server_negotiation.cc:408] Connection header received
1026 21:07:18.464953 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.464957 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.465019 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.465120 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:18.466036 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.466827 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.467728 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.467938 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.470765 (+ 2827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:18.470792 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:18.470795 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:18.470834 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:18.472674 (+ 1840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:18.473198 (+ 524us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:18.473201 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:18.473203 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:18.473252 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:18.473480 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:18.473484 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:18.473486 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:18.473685 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:18.473815 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.474065 (+ 250us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.474188 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 21:07:18.477605 11956 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:07:18.478137 11956 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:18.483554 12097 tablet_service.cc:1505] Processing CreateTablet for tablet 6972975a02ea43cabc673be2df8ec800 (DEFAULT_TABLE table=test-table [id=21d760c8b65c4c90a84d60c1da1ac560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.483932 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6972975a02ea43cabc673be2df8ec800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.483951 12381 tablet_service.cc:1505] Processing CreateTablet for tablet 6972975a02ea43cabc673be2df8ec800 (DEFAULT_TABLE table=test-table [id=21d760c8b65c4c90a84d60c1da1ac560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.484216 12381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6972975a02ea43cabc673be2df8ec800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.486459 12475 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Bootstrap starting.
I20251026 21:07:18.486470 12477 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Bootstrap starting.
I20251026 21:07:18.486645 12239 tablet_service.cc:1505] Processing CreateTablet for tablet 6972975a02ea43cabc673be2df8ec800 (DEFAULT_TABLE table=test-table [id=21d760c8b65c4c90a84d60c1da1ac560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.486930 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6972975a02ea43cabc673be2df8ec800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.487496 12477 tablet_bootstrap.cc:654] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.487610 12475 tablet_bootstrap.cc:654] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.489144 12475 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: No bootstrap required, opened a new log
I20251026 21:07:18.489238 12475 ts_tablet_manager.cc:1403] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:18.489344 12476 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Bootstrap starting.
I20251026 21:07:18.489776 12475 raft_consensus.cc:359] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.489908 12475 raft_consensus.cc:385] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.489946 12475 raft_consensus.cc:740] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488f7e81f00143059205a1776d98be29, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.490082 12475 consensus_queue.cc:260] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.490367 12475 ts_tablet_manager.cc:1434] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.490463 12476 tablet_bootstrap.cc:654] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.491611 12477 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: No bootstrap required, opened a new log
I20251026 21:07:18.491724 12477 ts_tablet_manager.cc:1403] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251026 21:07:18.492229 12477 raft_consensus.cc:359] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.492352 12477 raft_consensus.cc:385] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.492396 12477 raft_consensus.cc:740] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.492452 12476 tablet_bootstrap.cc:492] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: No bootstrap required, opened a new log
I20251026 21:07:18.492542 12476 ts_tablet_manager.cc:1403] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 21:07:18.492522 12477 consensus_queue.cc:260] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.493036 12476 raft_consensus.cc:359] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.493144 12476 raft_consensus.cc:385] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.493176 12476 raft_consensus.cc:740] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc42a6d0554610b48ec1d0eae343f6, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.493297 12477 ts_tablet_manager.cc:1434] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:18.493304 12476 consensus_queue.cc:260] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.493583 12476 ts_tablet_manager.cc:1434] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.532977 12485 raft_consensus.cc:493] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:18.533125 12485 raft_consensus.cc:515] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.533581 12485 leader_election.cc:290] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455), b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979)
I20251026 21:07:18.534140 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6972975a02ea43cabc673be2df8ec800" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29" is_pre_election: true
I20251026 21:07:18.534158 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6972975a02ea43cabc673be2df8ec800" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" is_pre_election: true
I20251026 21:07:18.534315 12259 raft_consensus.cc:2468] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.534315 12117 raft_consensus.cc:2468] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 0.
I20251026 21:07:18.534665 12335 leader_election.cc:304] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, b7fc42a6d0554610b48ec1d0eae343f6, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.534876 12485 raft_consensus.cc:2804] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:18.534960 12485 raft_consensus.cc:493] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:18.535017 12485 raft_consensus.cc:3060] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.536031 12485 raft_consensus.cc:515] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.536402 12485 leader_election.cc:290] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Requested vote from peers 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455), b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979)
I20251026 21:07:18.536794 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6972975a02ea43cabc673be2df8ec800" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
I20251026 21:07:18.536794 12117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6972975a02ea43cabc673be2df8ec800" candidate_uuid: "bff96662759e472197a53ac4b20f7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "488f7e81f00143059205a1776d98be29"
I20251026 21:07:18.536932 12117 raft_consensus.cc:3060] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.536932 12259 raft_consensus.cc:3060] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.538009 12117 raft_consensus.cc:2468] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.538015 12259 raft_consensus.cc:2468] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff96662759e472197a53ac4b20f7df0 in term 1.
I20251026 21:07:18.538342 12332 leader_election.cc:304] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.538533 12485 raft_consensus.cc:2804] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:18.538617 12485 raft_consensus.cc:697] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Becoming Leader. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Running, Role: LEADER
I20251026 21:07:18.538760 12485 consensus_queue.cc:237] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.540434 11958 catalog_manager.cc:5649] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bff96662759e472197a53ac4b20f7df0 (127.3.175.3). New cstate: current_term: 1 leader_uuid: "bff96662759e472197a53ac4b20f7df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } health_report { overall_health: HEALTHY } } }
I20251026 21:07:18.548346 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.546001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.3.175.3:44523)
1026 21:07:18.546138 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.546142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.546155 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:18.546236 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.546239 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.546285 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.546348 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.546633 (+ 285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.547094 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.547746 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.547879 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.547918 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.547987 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.548049 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.548147 (+ 98us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.548191 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 21:07:18.550367 12117 raft_consensus.cc:1275] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.550366 12259 raft_consensus.cc:1275] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Refusing update from remote peer bff96662759e472197a53ac4b20f7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.550796 12527 consensus_queue.cc:1048] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.550940 12485 consensus_queue.cc:1048] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.558154 11958 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.558343 11958 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.560407 11958 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Sending DeleteTablet for 3 replicas of tablet 6972975a02ea43cabc673be2df8ec800
I20251026 21:07:18.560962 12097 tablet_service.cc:1552] Processing DeleteTablet for tablet 6972975a02ea43cabc673be2df8ec800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:36560
I20251026 21:07:18.561079 12239 tablet_service.cc:1552] Processing DeleteTablet for tablet 6972975a02ea43cabc673be2df8ec800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52462
I20251026 21:07:18.561175 12497 tablet_replica.cc:333] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: stopping tablet replica
I20251026 21:07:18.561275 12498 tablet_replica.cc:333] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: stopping tablet replica
I20251026 21:07:18.561307 12497 raft_consensus.cc:2243] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.561393 12498 raft_consensus.cc:2243] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.561441 12497 raft_consensus.cc:2272] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.561506 12498 raft_consensus.cc:2272] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.561688 11958 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 21:07:18.561818 12381 tablet_service.cc:1552] Processing DeleteTablet for tablet 6972975a02ea43cabc673be2df8ec800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:49212
I20251026 21:07:18.561998 12499 tablet_replica.cc:333] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: stopping tablet replica
I20251026 21:07:18.562105 12499 raft_consensus.cc:2243] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 1 LEADER]: Raft consensus shutting down.
W20251026 21:07:18.562121 11958 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:18.562247 12498 ts_tablet_manager.cc:1916] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.562294 12499 raft_consensus.cc:2272] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.562997 12499 ts_tablet_manager.cc:1916] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.563746 12497 ts_tablet_manager.cc:1916] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.565352 12499 ts_tablet_manager.cc:1929] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.565423 12499 log.cc:1199] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/6972975a02ea43cabc673be2df8ec800
I20251026 21:07:18.565634 12498 ts_tablet_manager.cc:1929] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.565676 12499 ts_tablet_manager.cc:1950] T 6972975a02ea43cabc673be2df8ec800 P bff96662759e472197a53ac4b20f7df0: Deleting consensus metadata
I20251026 21:07:18.565701 12498 log.cc:1199] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/6972975a02ea43cabc673be2df8ec800
I20251026 21:07:18.565939 12498 ts_tablet_manager.cc:1950] T 6972975a02ea43cabc673be2df8ec800 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting consensus metadata
I20251026 21:07:18.565980 12497 ts_tablet_manager.cc:1929] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:18.566054 12497 log.cc:1199] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-0/wal/wals/6972975a02ea43cabc673be2df8ec800
I20251026 21:07:18.566335 12497 ts_tablet_manager.cc:1950] T 6972975a02ea43cabc673be2df8ec800 P 488f7e81f00143059205a1776d98be29: Deleting consensus metadata
I20251026 21:07:18.566435 11945 catalog_manager.cc:4985] TS bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523): tablet 6972975a02ea43cabc673be2df8ec800 (table test-table [id=21d760c8b65c4c90a84d60c1da1ac560]) successfully deleted
I20251026 21:07:18.567123 11942 catalog_manager.cc:4985] TS 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455): tablet 6972975a02ea43cabc673be2df8ec800 (table test-table [id=21d760c8b65c4c90a84d60c1da1ac560]) successfully deleted
I20251026 21:07:18.567348 12097 tablet_service.cc:1505] Processing CreateTablet for tablet b43700fd441c4856b8ccb302121d12f9 (DEFAULT_TABLE table=test-table [id=378d9492f0784dbdb74639c8128f275d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.567379 12239 tablet_service.cc:1505] Processing CreateTablet for tablet b43700fd441c4856b8ccb302121d12f9 (DEFAULT_TABLE table=test-table [id=378d9492f0784dbdb74639c8128f275d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.567610 12097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43700fd441c4856b8ccb302121d12f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.567610 12239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43700fd441c4856b8ccb302121d12f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.569677 12475 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29: Bootstrap starting.
I20251026 21:07:18.569726 12476 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Bootstrap starting.
I20251026 21:07:18.570509 12475 tablet_bootstrap.cc:654] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.570783 12476 tablet_bootstrap.cc:654] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.571734 12475 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29: No bootstrap required, opened a new log
I20251026 21:07:18.571825 12475 ts_tablet_manager.cc:1403] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:18.571892 12381 tablet_service.cc:1505] Processing CreateTablet for tablet b43700fd441c4856b8ccb302121d12f9 (DEFAULT_TABLE table=test-table [id=378d9492f0784dbdb74639c8128f275d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:18.572008 12476 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: No bootstrap required, opened a new log
I20251026 21:07:18.572083 12476 ts_tablet_manager.cc:1403] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 21:07:18.572188 12381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43700fd441c4856b8ccb302121d12f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:18.572292 12475 raft_consensus.cc:359] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.572392 12475 raft_consensus.cc:385] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.572432 12475 raft_consensus.cc:740] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488f7e81f00143059205a1776d98be29, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.572556 12475 consensus_queue.cc:260] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.572546 12476 raft_consensus.cc:359] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.572639 12476 raft_consensus.cc:385] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.572676 12476 raft_consensus.cc:740] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc42a6d0554610b48ec1d0eae343f6, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.572798 12476 consensus_queue.cc:260] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.572836 12475 ts_tablet_manager.cc:1434] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 21:07:18.573072 12476 ts_tablet_manager.cc:1434] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 21:07:18.573688 11945 catalog_manager.cc:4985] TS b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979): tablet 6972975a02ea43cabc673be2df8ec800 (table test-table [id=21d760c8b65c4c90a84d60c1da1ac560]) successfully deleted
I20251026 21:07:18.574515 12477 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Bootstrap starting.
I20251026 21:07:18.575532 12477 tablet_bootstrap.cc:654] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:18.576500 12477 tablet_bootstrap.cc:492] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: No bootstrap required, opened a new log
I20251026 21:07:18.576591 12477 ts_tablet_manager.cc:1403] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:18.576943 12477 raft_consensus.cc:359] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.577036 12477 raft_consensus.cc:385] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:18.577066 12477 raft_consensus.cc:740] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96662759e472197a53ac4b20f7df0, State: Initialized, Role: FOLLOWER
I20251026 21:07:18.577186 12477 consensus_queue.cc:260] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.577481 12477 ts_tablet_manager.cc:1434] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 21:07:18.621511 12522 raft_consensus.cc:493] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:18.621661 12522 raft_consensus.cc:515] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.622160 12522 leader_election.cc:290] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523)
I20251026 21:07:18.622604 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b43700fd441c4856b8ccb302121d12f9" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" is_pre_election: true
I20251026 21:07:18.622646 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b43700fd441c4856b8ccb302121d12f9" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff96662759e472197a53ac4b20f7df0" is_pre_election: true
I20251026 21:07:18.622762 12398 raft_consensus.cc:2468] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 0.
I20251026 21:07:18.622763 12259 raft_consensus.cc:2468] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 0.
I20251026 21:07:18.623087 12051 leader_election.cc:304] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, bff96662759e472197a53ac4b20f7df0; no voters:
I20251026 21:07:18.623272 12522 raft_consensus.cc:2804] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:18.623373 12522 raft_consensus.cc:493] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:18.623426 12522 raft_consensus.cc:3060] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.624341 12522 raft_consensus.cc:515] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.624665 12522 leader_election.cc:290] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 election: Requested vote from peers b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979), bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523)
I20251026 21:07:18.625046 12259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b43700fd441c4856b8ccb302121d12f9" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc42a6d0554610b48ec1d0eae343f6"
I20251026 21:07:18.625109 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b43700fd441c4856b8ccb302121d12f9" candidate_uuid: "488f7e81f00143059205a1776d98be29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff96662759e472197a53ac4b20f7df0"
I20251026 21:07:18.625166 12259 raft_consensus.cc:3060] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.625236 12398 raft_consensus.cc:3060] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:18.625989 12259 raft_consensus.cc:2468] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 1.
I20251026 21:07:18.626075 12398 raft_consensus.cc:2468] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 488f7e81f00143059205a1776d98be29 in term 1.
I20251026 21:07:18.626305 12051 leader_election.cc:304] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 488f7e81f00143059205a1776d98be29, b7fc42a6d0554610b48ec1d0eae343f6; no voters:
I20251026 21:07:18.626474 12522 raft_consensus.cc:2804] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:18.626570 12522 raft_consensus.cc:697] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [term 1 LEADER]: Becoming Leader. State: Replica: 488f7e81f00143059205a1776d98be29, State: Running, Role: LEADER
I20251026 21:07:18.626719 12522 consensus_queue.cc:237] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } }
I20251026 21:07:18.628482 11956 catalog_manager.cc:5649] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 488f7e81f00143059205a1776d98be29 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "488f7e81f00143059205a1776d98be29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "488f7e81f00143059205a1776d98be29" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 44455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 } health_report { overall_health: UNKNOWN } } }
I20251026 21:07:18.646611 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.644133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.3.175.1:44455)
1026 21:07:18.644268 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.644272 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.644287 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:18.644370 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.644374 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.644421 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.644485 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.644807 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.645307 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.645911 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.646083 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.646132 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.646214 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.646278 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.646411 (+ 133us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.646457 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251026 21:07:18.654393 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.652101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:46825 (local address 127.3.175.1:44455)
1026 21:07:18.652193 (+ 92us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:18.652198 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:18.652249 (+ 51us) server_negotiation.cc:408] Connection header received
1026 21:07:18.652371 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:18.652374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:18.652411 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:18.652474 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:18.652764 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.653214 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.653780 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.653946 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.653980 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.654037 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.654102 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:18.654215 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 21:07:18.654249 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 21:07:18.654393 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.652011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.3.175.1:46825)
1026 21:07:18.652174 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.652191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:18.652294 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:18.652486 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:18.652488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:18.652497 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 21:07:18.652671 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.652676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.653311 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:18.653315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:18.653687 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:18.653693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:18.653792 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:18.653820 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 21:07:18.654118 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 21:07:18.654132 (+ 14us) client_negotiation.cc:769] Sending connection context
1026 21:07:18.654200 (+ 68us) client_negotiation.cc:240] Negotiation successful
1026 21:07:18.654267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251026 21:07:18.660192 12259 raft_consensus.cc:1275] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Refusing update from remote peer 488f7e81f00143059205a1776d98be29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.660168 12398 raft_consensus.cc:1275] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Refusing update from remote peer 488f7e81f00143059205a1776d98be29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:18.660588 12515 consensus_queue.cc:1048] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc42a6d0554610b48ec1d0eae343f6" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.660714 12481 consensus_queue.cc:1048] T b43700fd441c4856b8ccb302121d12f9 P 488f7e81f00143059205a1776d98be29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bff96662759e472197a53ac4b20f7df0" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:18.696214 11957 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.696396 11957 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52190:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:18.698448 11957 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6dce224b11414773a482e966a0a9f227: Sending DeleteTablet for 3 replicas of tablet b43700fd441c4856b8ccb302121d12f9
I20251026 21:07:18.699216 12239 tablet_service.cc:1552] Processing DeleteTablet for tablet b43700fd441c4856b8ccb302121d12f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:52462
I20251026 21:07:18.699338 12381 tablet_service.cc:1552] Processing DeleteTablet for tablet b43700fd441c4856b8ccb302121d12f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:49212
I20251026 21:07:18.699462 12096 tablet_service.cc:1552] Processing DeleteTablet for tablet b43700fd441c4856b8ccb302121d12f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:18 UTC) from {username='oryx', principal='oryx/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:36560
I20251026 21:07:18.699532 12499 tablet_replica.cc:333] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: stopping tablet replica
I20251026 21:07:18.699653 12499 raft_consensus.cc:2243] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.699775 12499 raft_consensus.cc:2272] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.699800 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12032
I20251026 21:07:18.701645 12498 tablet_replica.cc:333] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: stopping tablet replica
I20251026 21:07:18.701786 12498 raft_consensus.cc:2243] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:18.701898 12498 raft_consensus.cc:2272] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:18.702231 12499 ts_tablet_manager.cc:1916] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.704335 12499 ts_tablet_manager.cc:1929] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:07:18.704398 12499 log.cc:1199] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/b43700fd441c4856b8ccb302121d12f9
I20251026 21:07:18.704608 12499 ts_tablet_manager.cc:1950] T b43700fd441c4856b8ccb302121d12f9 P bff96662759e472197a53ac4b20f7df0: Deleting consensus metadata
I20251026 21:07:18.705324 12498 ts_tablet_manager.cc:1916] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:18.705946 11945 catalog_manager.cc:4985] TS bff96662759e472197a53ac4b20f7df0 (127.3.175.3:44523): tablet b43700fd441c4856b8ccb302121d12f9 (table test-table [id=378d9492f0784dbdb74639c8128f275d]) successfully deleted
I20251026 21:07:18.707401 12498 ts_tablet_manager.cc:1929] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 21:07:18.707475 12498 log.cc:1199] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/b43700fd441c4856b8ccb302121d12f9
I20251026 21:07:18.707721 12498 ts_tablet_manager.cc:1950] T b43700fd441c4856b8ccb302121d12f9 P b7fc42a6d0554610b48ec1d0eae343f6: Deleting consensus metadata
I20251026 21:07:18.708447 11945 catalog_manager.cc:4985] TS b7fc42a6d0554610b48ec1d0eae343f6 (127.3.175.2:45979): tablet b43700fd441c4856b8ccb302121d12f9 (table test-table [id=378d9492f0784dbdb74639c8128f275d]) successfully deleted
W20251026 21:07:18.710240 11942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.175.1:44455 (error 108)
I20251026 21:07:18.711081 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12174
I20251026 21:07:18.712769 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:18.710961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:44455 (local address 127.0.0.1:36586)
1026 21:07:18.712535 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:18.712626 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:44455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
W20251026 21:07:18.712929 11942 catalog_manager.cc:4712] TS 488f7e81f00143059205a1776d98be29 (127.3.175.1:44455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b43700fd441c4856b8ccb302121d12f9: Network error: Client connection negotiation failed: client connection to 127.3.175.1:44455: connect: Connection refused (error 111)
I20251026 21:07:18.717506 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12316
I20251026 21:07:18.724478 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 11926
2025-10-26T21:07:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3913 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[12563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[12563](info): set up 2 sockets
Oct 26 21:07:18 dist-test-slave-th6x krb5kdc[12563](info): commencing operation
krb5kdc: starting...
W20251026 21:07:20.704079 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:20 dist-test-slave-th6x krb5kdc[12563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:20.845845 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43299
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:43299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:20.954691 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:20.954962 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:20.955055 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:20.958427 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:20.958503 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:20.958541 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:20.958577 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:20.958616 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:20.963021 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:43299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:20.964278 12579 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:20.965201 12579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:20.970836 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:20.970896 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:20.970911 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:20.971086 12579 server_base.cc:1047] running on GCE node
I20251026 21:07:20.971714 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:20.972690 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:20.973875 12579 hybrid_clock.cc:648] HybridClock initialized: now 1761512840973861 us; error 36 us; skew 500 ppm
W20251026 21:07:20.977228 12579 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-26T21:07:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2257 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:21 dist-test-slave-th6x krb5kdc[12592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:21 dist-test-slave-th6x krb5kdc[12592](info): set up 2 sockets
Oct 26 21:07:21 dist-test-slave-th6x krb5kdc[12592](info): commencing operation
krb5kdc: starting...
W20251026 21:07:22.976097 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:23.116379 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43813
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:23.225646 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:23.225899 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:23.225957 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:23.229302 12608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:23.229383 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:23.229406 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:23.229426 12608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:23.229444 12608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:23.233968 12608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:23.235189 12608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:23.236106 12608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:23.242216 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:23.242331 12608 server_base.cc:1047] running on GCE node
W20251026 21:07:23.242197 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:23.242189 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:23.242964 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:23.243971 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:23.245131 12608 hybrid_clock.cc:648] HybridClock initialized: now 1761512843245114 us; error 39 us; skew 500 ppm
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:23.247949 12608 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:07:23.249126 12608 webserver.cc:492] Webserver started at http://127.3.175.62:42141/ using document root <none> and password file <none>
I20251026 21:07:23.249702 12608 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:23.249756 12608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:23.249963 12608 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:23.251796 12608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "ed39a4e8546b4f1f9c04af73ae755da3"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "b2db5d472793d3c79a008c8c78a4db80"
server_key_iv: "20eb18af92275996bdc2f0fba1c01396"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.252313 12608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "ed39a4e8546b4f1f9c04af73ae755da3"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "b2db5d472793d3c79a008c8c78a4db80"
server_key_iv: "20eb18af92275996bdc2f0fba1c01396"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.255852 12608 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251026 21:07:23.258174 12623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.259243 12608 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:23.259425 12608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "ed39a4e8546b4f1f9c04af73ae755da3"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "b2db5d472793d3c79a008c8c78a4db80"
server_key_iv: "20eb18af92275996bdc2f0fba1c01396"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.259534 12608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:23.277005 12608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:23.285429 12608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:23.285647 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:23.293363 12675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:43813 every 8 connection(s)
I20251026 21:07:23.293380 12608 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:43813
I20251026 21:07:23.294462 12608 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:23.297385 12676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:23.303025 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12608
I20251026 21:07:23.303169 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:23.303464 3772 external_mini_cluster.cc:1468] Setting key 98f1776d0db9f9edb02aa6a6528ef1aa
I20251026 21:07:23.303504 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Bootstrap starting.
I20251026 21:07:23.306097 12676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:23.307013 12676 log.cc:826] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:23.309186 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: No bootstrap required, opened a new log
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:23.312603 12676 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:23.312793 12676 raft_consensus.cc:385] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:23.312855 12676 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed39a4e8546b4f1f9c04af73ae755da3, State: Initialized, Role: FOLLOWER
I20251026 21:07:23.313323 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:23.313498 12676 raft_consensus.cc:399] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:23.313575 12676 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:23.313685 12676 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:23.314551 12676 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:23.314991 12676 leader_election.cc:304] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed39a4e8546b4f1f9c04af73ae755da3; no voters:
I20251026 21:07:23.315433 12676 leader_election.cc:290] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:23.315793 12681 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:23.316069 12681 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 LEADER]: Becoming Leader. State: Replica: ed39a4e8546b4f1f9c04af73ae755da3, State: Running, Role: LEADER
I20251026 21:07:23.316386 12681 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:23.316707 12676 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:23.317718 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:23.304756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.3.175.62:43813)
1026 21:07:23.305179 (+ 423us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:23.305189 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:23.305225 (+ 36us) server_negotiation.cc:408] Connection header received
1026 21:07:23.305810 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:23.305827 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:23.306129 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:23.306455 (+ 326us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:23.307520 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.308474 (+ 954us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:23.309327 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.309558 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:23.312222 (+ 2664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:23.312243 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:23.312256 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:23.312292 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:23.314374 (+ 2082us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.314877 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.314882 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.314888 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.314964 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.315221 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.315223 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.315225 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.315662 (+ 437us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:23.316613 (+ 951us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:23.316772 (+ 159us) server_negotiation.cc:300] Negotiation successful
1026 21:07:23.317030 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
I20251026 21:07:23.318001 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed39a4e8546b4f1f9c04af73ae755da3. Latest consensus state: current_term: 1 leader_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } } }
I20251026 21:07:23.318115 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:23.318500 12690 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:23.318470 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } } }
I20251026 21:07:23.318553 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:23.321457 12690 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:23.327014 12690 catalog_manager.cc:1357] Generated new cluster ID: a272c1dc5f4743189510c6f3edb947b2
I20251026 21:07:23.327095 12690 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:23.343555 12690 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:23.344600 12690 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:23.352777 12690 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Generated new TSK 0
I20251026 21:07:23.353715 12690 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:23.414685 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43813
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:23.527376 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:23.527628 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:23.527693 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:23.531076 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:23.531157 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:23.531240 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:23.535728 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:23.536928 12704 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:23.537930 12704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:23.545043 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:23.545044 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:23.545047 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:23.545343 12704 server_base.cc:1047] running on GCE node
I20251026 21:07:23.545750 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:23.546377 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:23.547534 12704 hybrid_clock.cc:648] HybridClock initialized: now 1761512843547510 us; error 39 us; skew 500 ppm
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:23.550211 12704 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:07:23.551349 12704 webserver.cc:492] Webserver started at http://127.3.175.1:45317/ using document root <none> and password file <none>
I20251026 21:07:23.551929 12704 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:23.551985 12704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:23.552177 12704 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:23.553926 12704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "760b6913e3f4413ebfac0e9b7c460e8a"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "a6bd1144833bed423f00d17c2e2a8619"
server_key_iv: "17212df4285cf89ae1316c3004b240ca"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.554409 12704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "760b6913e3f4413ebfac0e9b7c460e8a"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "a6bd1144833bed423f00d17c2e2a8619"
server_key_iv: "17212df4285cf89ae1316c3004b240ca"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.557904 12704 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251026 21:07:23.560380 12719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.561496 12704 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:23.561626 12704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "760b6913e3f4413ebfac0e9b7c460e8a"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "a6bd1144833bed423f00d17c2e2a8619"
server_key_iv: "17212df4285cf89ae1316c3004b240ca"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.561738 12704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:23.578606 12704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:23.581647 12704 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:23.581847 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:23.582440 12704 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:23.583384 12704 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:23.583434 12704 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.583495 12704 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:23.583527 12704 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.592702 12704 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:41241
I20251026 21:07:23.592731 12832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:41241 every 8 connection(s)
I20251026 21:07:23.593719 12704 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:23.601082 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12704
I20251026 21:07:23.601251 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:23.601537 3772 external_mini_cluster.cc:1468] Setting key 8c973b6ea911c768152afb560400ac33
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:23.606761 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:23.595567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:58349 (local address 127.3.175.62:43813)
1026 21:07:23.595712 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:23.595716 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:23.596416 (+ 700us) server_negotiation.cc:408] Connection header received
1026 21:07:23.597237 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:23.597240 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:23.597291 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:23.597366 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:23.599064 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.599628 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:23.600302 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.600439 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:23.603095 (+ 2656us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:23.603112 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:23.603114 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:23.603150 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:23.604821 (+ 1671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.605331 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.605335 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.605336 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.605377 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.605644 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.605647 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.605649 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.605804 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:23.605899 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:23.606451 (+ 552us) server_negotiation.cc:300] Negotiation successful
1026 21:07:23.606582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 21:07:23.607494 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:23.595830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:58349)
1026 21:07:23.596264 (+ 434us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:23.596301 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:23.597053 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:23.597521 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:23.597530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:23.597950 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:23.598923 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:23.598937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.599749 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:23.599752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:23.600182 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:23.600190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.600398 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:23.601067 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:23.601114 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:23.602960 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:23.604954 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:23.604959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:23.604970 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:23.605225 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:23.605460 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:23.605464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:23.605465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:23.605557 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:23.605900 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:23.605910 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:23.606202 (+ 292us) client_negotiation.cc:769] Sending connection context
1026 21:07:23.606407 (+ 205us) client_negotiation.cc:240] Negotiation successful
1026 21:07:23.606693 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20251026 21:07:23.608906 12833 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43813
I20251026 21:07:23.609156 12833 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:23.609678 12833 heartbeater.cc:507] Master 127.3.175.62:43813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:07:23.611284 12640 ts_manager.cc:194] Registered new tserver with Master: 760b6913e3f4413ebfac0e9b7c460e8a (127.3.175.1:41241)
I20251026 21:07:23.612560 12640 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:58349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:23.652459 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43813
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:23.762357 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:23.762627 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:23.762705 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:23.766170 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:23.766253 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:23.766373 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:23.770852 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:23.772167 12840 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:23.773038 12840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:23.779718 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:23.779915 12840 server_base.cc:1047] running on GCE node
W20251026 21:07:23.779714 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:23.779714 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:23.780424 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:23.780963 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:23.782130 12840 hybrid_clock.cc:648] HybridClock initialized: now 1761512843782118 us; error 28 us; skew 500 ppm
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:23.785089 12840 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:07:23.786293 12840 webserver.cc:492] Webserver started at http://127.3.175.2:37793/ using document root <none> and password file <none>
I20251026 21:07:23.786969 12840 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:23.787031 12840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:23.787250 12840 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:23.789191 12840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "e79e06047d4c4cca809cd26d02e5f445"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "218d977743e700387478695ea1c8a8a7"
server_key_iv: "b8d010d27088efacca1597486d5a6d16"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.789722 12840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "e79e06047d4c4cca809cd26d02e5f445"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "218d977743e700387478695ea1c8a8a7"
server_key_iv: "b8d010d27088efacca1597486d5a6d16"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.793494 12840 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20251026 21:07:23.795902 12855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.797008 12840 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:23.797134 12840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "e79e06047d4c4cca809cd26d02e5f445"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "218d977743e700387478695ea1c8a8a7"
server_key_iv: "b8d010d27088efacca1597486d5a6d16"
server_key_version: "encryptionkey@0"
I20251026 21:07:23.797240 12840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:23.818792 12840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:23.821704 12840 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:23.821902 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:23.822471 12840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:23.823418 12840 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:23.823468 12840 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.823529 12840 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:23.823558 12840 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:23.832471 12840 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:32779
I20251026 21:07:23.832504 12968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:32779 every 8 connection(s)
I20251026 21:07:23.833431 12840 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:23.838835 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12840
I20251026 21:07:23.838970 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:23.839254 3772 external_mini_cluster.cc:1468] Setting key 0ba7bd5d69cd2a125e5243748be2828d
Oct 26 21:07:23 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:23.847622 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:23.835235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:37845 (local address 127.3.175.62:43813)
1026 21:07:23.835437 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:23.835442 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:23.836124 (+ 682us) server_negotiation.cc:408] Connection header received
1026 21:07:23.837046 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:23.837049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:23.837112 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:23.837199 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:23.839156 (+ 1957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.839740 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:23.840450 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.840629 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:23.843434 (+ 2805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:23.843459 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:23.843466 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:23.843502 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:23.845287 (+ 1785us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.845960 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.845963 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.845964 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.846014 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:23.846376 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:23.846378 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:23.846379 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:23.846551 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:23.846624 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:23.847365 (+ 741us) server_negotiation.cc:300] Negotiation successful
1026 21:07:23.847477 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20251026 21:07:23.848675 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:23.835495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:37845)
1026 21:07:23.835951 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:23.835989 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:23.836828 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:23.837375 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:23.837383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:23.837790 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:23.838914 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:23.838936 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.839863 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:23.839867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:23.840331 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:23.840340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:23.840585 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:23.841263 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:23.841304 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:23.843222 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:23.845423 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:23.845434 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:23.845450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:23.845847 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:23.846122 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:23.846128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:23.846132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:23.846280 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:23.846655 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:23.846667 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:23.847044 (+ 377us) client_negotiation.cc:769] Sending connection context
1026 21:07:23.847351 (+ 307us) client_negotiation.cc:240] Negotiation successful
1026 21:07:23.847684 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:07:23.849982 12969 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43813
I20251026 21:07:23.850296 12969 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:23.850931 12969 heartbeater.cc:507] Master 127.3.175.62:43813 requested a full tablet report, sending...
I20251026 21:07:23.852144 12640 ts_manager.cc:194] Registered new tserver with Master: e79e06047d4c4cca809cd26d02e5f445 (127.3.175.2:32779)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:07:23.852806 12640 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:37845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:23.890471 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43813
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:24.000734 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:24.000998 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:24.001060 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:24.004490 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:24.004575 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:24.004659 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:24.009096 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:24.010238 12976 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:24.011082 12976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:24.018111 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.018087 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.018224 12976 server_base.cc:1047] running on GCE node
W20251026 21:07:24.018090 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.018808 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:24.019410 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:24.020563 12976 hybrid_clock.cc:648] HybridClock initialized: now 1761512844020553 us; error 27 us; skew 500 ppm
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:24.023550 12976 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:07:24.024654 12976 webserver.cc:492] Webserver started at http://127.3.175.3:41947/ using document root <none> and password file <none>
I20251026 21:07:24.025211 12976 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:24.025259 12976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:24.025424 12976 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:24.027171 12976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "6219dad811fe48c5908e126a2d79bacb"
format_stamp: "Formatted at 2025-10-26 21:07:24 on dist-test-slave-th6x"
server_key: "a034aa9abf36a1eb2cf66b75a3150334"
server_key_iv: "6dba6ebed7822b29e2a66465cd6fc3ab"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.027694 12976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "6219dad811fe48c5908e126a2d79bacb"
format_stamp: "Formatted at 2025-10-26 21:07:24 on dist-test-slave-th6x"
server_key: "a034aa9abf36a1eb2cf66b75a3150334"
server_key_iv: "6dba6ebed7822b29e2a66465cd6fc3ab"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.031188 12976 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251026 21:07:24.033598 12991 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.034636 12976 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:24.034741 12976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "6219dad811fe48c5908e126a2d79bacb"
format_stamp: "Formatted at 2025-10-26 21:07:24 on dist-test-slave-th6x"
server_key: "a034aa9abf36a1eb2cf66b75a3150334"
server_key_iv: "6dba6ebed7822b29e2a66465cd6fc3ab"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.034848 12976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:24.055660 12976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:24.058557 12976 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:24.058748 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:24.059378 12976 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:24.060353 12976 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:24.060401 12976 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.060446 12976 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:24.060482 12976 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.070056 12976 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:33963
I20251026 21:07:24.070077 13104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:33963 every 8 connection(s)
I20251026 21:07:24.071105 12976 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:24.076933 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 12976
I20251026 21:07:24.077064 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:24.077339 3772 external_mini_cluster.cc:1468] Setting key 8a1e80b0951c8bc106dc415f893f291e
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:24.083917 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.072956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:40667 (local address 127.3.175.62:43813)
1026 21:07:24.073095 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.073099 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.073813 (+ 714us) server_negotiation.cc:408] Connection header received
1026 21:07:24.074677 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.074681 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.074744 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.074829 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.076624 (+ 1795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.077375 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.078126 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.078315 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.080763 (+ 2448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:24.080780 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:24.080782 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:24.080809 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:24.082165 (+ 1356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:24.082627 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:24.082630 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:24.082631 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:24.082669 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:24.082872 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:24.082874 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:24.082875 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:24.083034 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:24.083105 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:24.083646 (+ 541us) server_negotiation.cc:300] Negotiation successful
1026 21:07:24.083775 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:07:24.084661 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.073230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:40667)
1026 21:07:24.073671 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.073708 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.074475 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.075012 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.075020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.075492 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.076456 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.076471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.077515 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.077519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.078004 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.078014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.078250 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.078985 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.079012 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.080616 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:24.082280 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:24.082286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:24.082297 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:24.082543 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:24.082728 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:24.082730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:24.082731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:24.082812 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:24.083110 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:24.083115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:24.083405 (+ 290us) client_negotiation.cc:769] Sending connection context
1026 21:07:24.083590 (+ 185us) client_negotiation.cc:240] Negotiation successful
1026 21:07:24.083852 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20251026 21:07:24.085840 13105 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43813
I20251026 21:07:24.086095 13105 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:24.086647 13105 heartbeater.cc:507] Master 127.3.175.62:43813 requested a full tablet report, sending...
I20251026 21:07:24.087698 12640 ts_manager.cc:194] Registered new tserver with Master: 6219dad811fe48c5908e126a2d79bacb (127.3.175.3:33963)
I20251026 21:07:24.088274 12640 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:40667
I20251026 21:07:24.091187 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.1.keytab.
Entry for principal oryx/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.1.keytab.
WARNING: no policy specified for oryx/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.2.keytab.
Entry for principal oryx/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.2.keytab.
WARNING: no policy specified for oryx/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.3.keytab.
Entry for principal oryx/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.3.keytab.
I20251026 21:07:24.162225 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12704
I20251026 21:07:24.168206 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12840
I20251026 21:07:24.173879 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12976
I20251026 21:07:24.179917 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 12608
I20251026 21:07:24.186633 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43813
--webserver_interface=127.3.175.62
--webserver_port=42141
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:24.297632 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:24.297916 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:24.298017 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:24.301474 13114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:24.301563 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:24.301605 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:24.301637 13114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:24.301677 13114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:24.306326 13114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=42141
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:24.307576 13114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:24.308490 13114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:24.314682 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.314682 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.314736 13114 server_base.cc:1047] running on GCE node
W20251026 21:07:24.314685 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.315497 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:24.316427 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:24.317580 13114 hybrid_clock.cc:648] HybridClock initialized: now 1761512844317569 us; error 27 us; skew 500 ppm
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:24.320451 13114 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:07:24.321605 13114 webserver.cc:492] Webserver started at http://127.3.175.62:42141/ using document root <none> and password file <none>
I20251026 21:07:24.322188 13114 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:24.322258 13114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:24.326011 13114 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:24.328121 13129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.329192 13114 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:24.329332 13114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "ed39a4e8546b4f1f9c04af73ae755da3"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "b2db5d472793d3c79a008c8c78a4db80"
server_key_iv: "20eb18af92275996bdc2f0fba1c01396"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.329771 13114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:24.348405 13114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:24.351591 13114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:24.351781 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:24.359431 13114 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:43813
I20251026 21:07:24.359432 13181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:43813 every 8 connection(s)
I20251026 21:07:24.360584 13114 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:24.362540 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13114
I20251026 21:07:24.363068 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:41241
--local_ip_for_outbound_sockets=127.3.175.1
--tserver_master_addrs=127.3.175.62:43813
--webserver_port=45317
--webserver_interface=127.3.175.1
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.175.1
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 21:07:24.366019 13182 sys_catalog.cc:263] Verifying existing consensus state
I20251026 21:07:24.367733 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Bootstrap starting.
I20251026 21:07:24.383837 13182 log.cc:826] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:24.390077 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 21:07:24.390458 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Bootstrap complete.
I20251026 21:07:24.393980 13182 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:24.394296 13182 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed39a4e8546b4f1f9c04af73ae755da3, State: Initialized, Role: FOLLOWER
I20251026 21:07:24.394771 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:24.394891 13182 raft_consensus.cc:399] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:24.394958 13182 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:24.395047 13182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 1 FOLLOWER]: Advancing to term 2
I20251026 21:07:24.396634 13182 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:24.396971 13182 leader_election.cc:304] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed39a4e8546b4f1f9c04af73ae755da3; no voters:
I20251026 21:07:24.397264 13182 leader_election.cc:290] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 21:07:24.397373 13186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 21:07:24.397575 13186 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [term 2 LEADER]: Becoming Leader. State: Replica: ed39a4e8546b4f1f9c04af73ae755da3, State: Running, Role: LEADER
I20251026 21:07:24.397859 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } }
I20251026 21:07:24.398418 13182 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:24.398811 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed39a4e8546b4f1f9c04af73ae755da3. Latest consensus state: current_term: 2 leader_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } } }
I20251026 21:07:24.398916 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:24.399303 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed39a4e8546b4f1f9c04af73ae755da3" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43813 } } }
I20251026 21:07:24.399420 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:24.399271 13190 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:24.402733 13190 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:24.403632 13190 catalog_manager.cc:1269] Loaded cluster ID: a272c1dc5f4743189510c6f3edb947b2
I20251026 21:07:24.403719 13190 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:24.406188 13190 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:24.406903 13190 catalog_manager.cc:6033] T 00000000000000000000000000000000 P ed39a4e8546b4f1f9c04af73ae755da3: Loaded TSK: 0
I20251026 21:07:24.408219 13190 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 21:07:24.477715 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:24.477991 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:24.478089 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:24.481619 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:24.481712 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:24.481839 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:24.486393 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.1.keytab
--principal=oryx/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=45317
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:24.487663 13184 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:24.488538 13184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:24.495445 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.495453 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.495548 13184 server_base.cc:1047] running on GCE node
W20251026 21:07:24.495756 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.496212 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:24.496914 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:24.498093 13184 hybrid_clock.cc:648] HybridClock initialized: now 1761512844498060 us; error 47 us; skew 500 ppm
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:24.501533 13184 init.cc:377] Logged in from keytab as oryx/127.3.175.1@KRBTEST.COM (short username oryx)
I20251026 21:07:24.502915 13184 webserver.cc:492] Webserver started at http://127.3.175.1:45317/ using document root <none> and password file <none>
I20251026 21:07:24.503717 13184 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:24.503810 13184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:24.508522 13184 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20251026 21:07:24.510859 13218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.512097 13184 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:07:24.512235 13184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "760b6913e3f4413ebfac0e9b7c460e8a"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "a6bd1144833bed423f00d17c2e2a8619"
server_key_iv: "17212df4285cf89ae1316c3004b240ca"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.512763 13184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:24.530504 13184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:24.534214 13184 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:24.534423 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:24.535121 13184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:24.536285 13184 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:24.536347 13184 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.536414 13184 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:24.536444 13184 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.548600 13184 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:41241
I20251026 21:07:24.548615 13331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:41241 every 8 connection(s)
I20251026 21:07:24.549965 13184 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:24.559549 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13184
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.560035 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:32779
--local_ip_for_outbound_sockets=127.3.175.2
--tserver_master_addrs=127.3.175.62:43813
--webserver_port=37793
--webserver_interface=127.3.175.2
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.175.2
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.562708 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.552390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:41945)
1026 21:07:24.552857 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.552902 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.553676 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.554715 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.554727 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.555207 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.556203 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.556219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.557355 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.557358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.557774 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.557782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.558030 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.558835 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.558860 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.561885 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
W20251026 21:07:24.562881 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.552176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:41945 (local address 127.3.175.62:43813)
1026 21:07:24.552588 (+ 412us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.552598 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.553027 (+ 429us) server_negotiation.cc:408] Connection header received
1026 21:07:24.554064 (+ 1037us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.554074 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.554346 (+ 272us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.554633 (+ 287us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.556375 (+ 1742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.557213 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.557892 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.558148 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.562169 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:41945: BlockingRecv error: recv got EOF from 127.3.175.1:41945 (error 108)
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":102,"threads_started":1}
W20251026 21:07:24.563462 13332 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.175.62:43813: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.570436 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.564009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:40667)
1026 21:07:24.564262 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.564277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.564369 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.564762 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.564768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.564981 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.565296 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.565309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.566367 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.566371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.567034 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.567047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.567200 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.567880 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.567901 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.570286 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169}
W20251026 21:07:24.570601 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.564139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:40667 (local address 127.3.175.62:43813)
1026 21:07:24.564515 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.564519 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.564532 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:24.564573 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.564576 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.564621 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.564696 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.565440 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.566239 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.567184 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.567838 (+ 654us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.570452 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:40667: BlockingRecv error: recv got EOF from 127.3.175.1:40667 (error 108)
Metrics: {"server-negotiator.queue_time_us":300}
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.577304 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.571340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:39185)
1026 21:07:24.571555 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.571574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.571670 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.572102 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.572106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.572329 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.572594 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.572602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.573633 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.573639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.574248 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.574258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.574382 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.574968 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.574984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.577149 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20251026 21:07:24.577484 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.571418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:39185 (local address 127.3.175.62:43813)
1026 21:07:24.571825 (+ 407us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.571829 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.571841 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:24.571903 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.571906 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.571968 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.572051 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.572730 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.573496 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.574376 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.574552 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.577356 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:39185: BlockingRecv error: recv got EOF from 127.3.175.1:39185 (error 108)
Metrics: {"server-negotiator.queue_time_us":330}
W20251026 21:07:24.577831 13332 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 21:07:24.676874 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:24.677135 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:24.677237 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:24.680713 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:24.680809 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:24.680943 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:24.685501 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.2.keytab
--principal=oryx/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:32779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=37793
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:24.686743 13336 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:24.687682 13336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:24.694792 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.694824 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.694792 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.695104 13336 server_base.cc:1047] running on GCE node
I20251026 21:07:24.695569 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:24.696174 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:24.697331 13336 hybrid_clock.cc:648] HybridClock initialized: now 1761512844697317 us; error 35 us; skew 500 ppm
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:24.700345 13336 init.cc:377] Logged in from keytab as oryx/127.3.175.2@KRBTEST.COM (short username oryx)
I20251026 21:07:24.701565 13336 webserver.cc:492] Webserver started at http://127.3.175.2:37793/ using document root <none> and password file <none>
I20251026 21:07:24.702170 13336 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:24.702235 13336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:24.706070 13336 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:24.708149 13351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.709205 13336 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:07:24.709331 13336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "e79e06047d4c4cca809cd26d02e5f445"
format_stamp: "Formatted at 2025-10-26 21:07:23 on dist-test-slave-th6x"
server_key: "218d977743e700387478695ea1c8a8a7"
server_key_iv: "b8d010d27088efacca1597486d5a6d16"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.709782 13336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:24.726246 13336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:24.729218 13336 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:24.729409 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:24.729970 13336 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:24.730885 13336 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:24.730942 13336 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.731002 13336 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:24.731027 13336 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.740648 13336 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:32779
I20251026 21:07:24.740665 13464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:32779 every 8 connection(s)
I20251026 21:07:24.741698 13336 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:24.746670 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13336
I20251026 21:07:24.747104 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:33963
--local_ip_for_outbound_sockets=127.3.175.3
--tserver_master_addrs=127.3.175.62:43813
--webserver_port=41947
--webserver_interface=127.3.175.3
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.175.3
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
W20251026 21:07:24.752537 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.743588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:55797 (local address 127.3.175.62:43813)
1026 21:07:24.743800 (+ 212us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.743805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.744487 (+ 682us) server_negotiation.cc:408] Connection header received
1026 21:07:24.745309 (+ 822us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.745313 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.745370 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.745445 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.747216 (+ 1771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.748035 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.748753 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.748953 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.752407 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:55797: BlockingRecv error: recv got EOF from 127.3.175.2:55797 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20251026 21:07:24.753098 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.743863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:55797)
1026 21:07:24.744333 (+ 470us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.744370 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.745108 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.745600 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.745609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.746011 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.747028 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.747044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.748185 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.748188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.748605 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.748613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.748877 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.749619 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.749645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.752243 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
W20251026 21:07:24.753732 13465 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.175.62:43813: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.759744 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.754263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:58645)
1026 21:07:24.754490 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.754507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.754616 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.754895 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.754902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.755103 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.755429 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.755439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.756199 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.756204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.756849 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.756859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.756979 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.757605 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.757628 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.759609 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20251026 21:07:24.759842 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.754345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:58645 (local address 127.3.175.62:43813)
1026 21:07:24.754467 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.754470 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.754558 (+ 88us) server_negotiation.cc:408] Connection header received
1026 21:07:24.754728 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.754731 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.754780 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.754856 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.755557 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.756078 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.757051 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.757218 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.759740 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:58645: BlockingRecv error: recv got EOF from 127.3.175.2:58645 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.764883 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.760441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:41223)
1026 21:07:24.760580 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.760596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.760681 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.760919 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.760922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.761069 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.761238 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.761244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.761928 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.761934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.762292 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.762299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.762401 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.762966 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.762979 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.764734 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20251026 21:07:24.764981 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.760524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:41223 (local address 127.3.175.62:43813)
1026 21:07:24.760719 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.760725 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.760740 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:24.760776 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.760779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.760823 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.760902 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.761350 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.761825 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.762397 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.762596 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.764860 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:41223: BlockingRecv error: recv got EOF from 127.3.175.2:41223 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20251026 21:07:24.765249 13465 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 21:07:24.862399 13468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:24.862681 13468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:24.862740 13468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:24.866184 13468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:24.866276 13468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:24.866361 13468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:24.870862 13468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/krb5kdc/oryx_127.3.175.3.keytab
--principal=oryx/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=41947
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:24.872109 13468 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:24.873025 13468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:24.879904 13473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.879796 13476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:24.879789 13474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:24.880153 13468 server_base.cc:1047] running on GCE node
I20251026 21:07:24.880621 13468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:24.881229 13468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:24.882400 13468 hybrid_clock.cc:648] HybridClock initialized: now 1761512844882369 us; error 49 us; skew 500 ppm
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512844, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:24.885255 13468 init.cc:377] Logged in from keytab as oryx/127.3.175.3@KRBTEST.COM (short username oryx)
I20251026 21:07:24.886397 13468 webserver.cc:492] Webserver started at http://127.3.175.3:41947/ using document root <none> and password file <none>
I20251026 21:07:24.886989 13468 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:24.887059 13468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:24.890743 13468 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:24.892849 13483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.894030 13468 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 21:07:24.894179 13468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "6219dad811fe48c5908e126a2d79bacb"
format_stamp: "Formatted at 2025-10-26 21:07:24 on dist-test-slave-th6x"
server_key: "a034aa9abf36a1eb2cf66b75a3150334"
server_key_iv: "6dba6ebed7822b29e2a66465cd6fc3ab"
server_key_version: "encryptionkey@0"
I20251026 21:07:24.894649 13468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:24.909799 13468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:24.912792 13468 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:24.912961 13468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:24.913511 13468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:24.914534 13468 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:24.914589 13468 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.914652 13468 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:24.914683 13468 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:24.924459 13468 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:33963
I20251026 21:07:24.924481 13596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:33963 every 8 connection(s)
I20251026 21:07:24.925571 13468 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.934073 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13468
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
W20251026 21:07:24.935943 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.927516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:52655 (local address 127.3.175.62:43813)
1026 21:07:24.927661 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.927664 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.928542 (+ 878us) server_negotiation.cc:408] Connection header received
1026 21:07:24.929348 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.929351 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.929402 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.929480 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.931150 (+ 1670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.931698 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.932336 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.932463 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.935782 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:52655: BlockingRecv error: recv got EOF from 127.3.175.3:52655 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:07:24.936368 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.927803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:52655)
1026 21:07:24.928385 (+ 582us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.928424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.929170 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.929632 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.929639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.930045 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.931013 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.931028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.931811 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.931813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.932237 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.932244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.932424 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.933014 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.933036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.935644 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":116,"threads_started":1}
W20251026 21:07:24.936795 13597 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.175.62:43813: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.942304 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.937212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:37639)
1026 21:07:24.937399 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.937414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.937490 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.937723 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.937726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.937907 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.938135 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.938148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.938839 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.938846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.939501 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.939513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.939646 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.940172 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.940190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.942151 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20251026 21:07:24.942377 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.937318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:37639 (local address 127.3.175.62:43813)
1026 21:07:24.937441 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.937444 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.937454 (+ 10us) server_negotiation.cc:408] Connection header received
1026 21:07:24.937590 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.937592 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.937631 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.937686 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.938238 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.938730 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.939619 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.939785 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.942269 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:37639: BlockingRecv error: recv got EOF from 127.3.175.3:37639 (error 108)
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":3456}
I20251026 21:07:24.944408 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.935125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.3.175.62:43813)
1026 21:07:24.935440 (+ 315us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.935444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.935458 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:24.935497 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.935500 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.935546 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.935634 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.936707 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.937308 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.938180 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.938366 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.939721 (+ 1355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:24.939744 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:24.939762 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:24.939814 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:24.941991 (+ 2177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:24.942672 (+ 681us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:24.942678 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:24.942683 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:24.942772 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:24.943055 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:24.943059 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:24.943061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:24.943531 (+ 470us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:24.943777 (+ 246us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:24.944028 (+ 251us) server_negotiation.cc:300] Negotiation successful
1026 21:07:24.944283 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:24 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:24.947717 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:24.942816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:54817)
1026 21:07:24.943123 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:24.943138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:24.943220 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:24.943467 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:24.943470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:24.943636 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:24.943864 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.943872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.944677 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.944681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:24.945059 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:24.945067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.945184 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.945720 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:24.945734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:24.947610 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224}
W20251026 21:07:24.947883 13334 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:24.942912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:54817 (local address 127.3.175.62:43813)
1026 21:07:24.943040 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:24.943043 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:24.943190 (+ 147us) server_negotiation.cc:408] Connection header received
1026 21:07:24.943320 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:24.943324 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:24.943370 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:24.943433 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:24.943988 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.944553 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:24.945178 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:24.945379 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:24.947740 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:54817: BlockingRecv error: recv got EOF from 127.3.175.3:54817 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20251026 21:07:24.947913 13597 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:25.583381 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:25.578387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:46775)
1026 21:07:25.578685 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:25.578704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:25.578854 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:25.579108 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:25.579111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:25.579379 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:25.579584 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.579590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.580309 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.580312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:25.580716 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.580723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.580825 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.581436 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:25.581450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:25.583218 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
W20251026 21:07:25.583523 13603 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:25.578544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:46775 (local address 127.3.175.62:43813)
1026 21:07:25.578817 (+ 273us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:25.578820 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:25.578833 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:25.578942 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:25.578945 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:25.578993 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:25.579068 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:25.579688 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.580211 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.580812 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.580940 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.583401 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:46775: BlockingRecv error: recv got EOF from 127.3.175.1:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:25.771273 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:25.766173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:40887)
1026 21:07:25.766486 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:25.766503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:25.766631 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:25.766918 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:25.766922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:25.767121 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:25.767427 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.767438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.768201 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.768207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:25.768588 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.768594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.768702 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.769367 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:25.769381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:25.771146 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
W20251026 21:07:25.771442 13603 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:25.766358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:40887 (local address 127.3.175.62:43813)
1026 21:07:25.766535 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:25.766542 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:25.766564 (+ 22us) server_negotiation.cc:408] Connection header received
1026 21:07:25.766750 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:25.766753 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:25.766801 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:25.766875 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:25.767552 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.768082 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.768697 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.768917 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.771336 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:40887: BlockingRecv error: recv got EOF from 127.3.175.2:40887 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:25 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:25.953665 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:25.948628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:53743)
1026 21:07:25.948969 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:25.948983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:25.949095 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:25.949349 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:25.949351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:25.949569 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:25.949786 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.949831 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.950543 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.950546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:25.950982 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:25.950990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.951090 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.951690 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:25.951704 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:25.953532 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
W20251026 21:07:25.953789 13603 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:25.948661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:53743 (local address 127.3.175.62:43813)
1026 21:07:25.948843 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:25.948847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:25.949061 (+ 214us) server_negotiation.cc:408] Connection header received
1026 21:07:25.949190 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:25.949193 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:25.949243 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:25.949318 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:25.949938 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.950435 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:25.951074 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:25.951211 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:25.953683 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:53743: BlockingRecv error: recv got EOF from 127.3.175.3:53743 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:26.589277 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:26.584295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:47719)
1026 21:07:26.584619 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:26.584634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:26.584729 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:26.585018 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:26.585021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:26.585213 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:26.585424 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.585430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.586159 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.586162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:26.586565 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.586572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.586670 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.587319 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:26.587333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:26.589144 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":109,"threads_started":1}
W20251026 21:07:26.589489 13607 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:26.584424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:47719 (local address 127.3.175.62:43813)
1026 21:07:26.584658 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:26.584661 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:26.584678 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:07:26.584828 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:26.584830 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:26.584879 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:26.584960 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:26.585527 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.586045 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.586663 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.586801 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.589337 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:47719: BlockingRecv error: recv got EOF from 127.3.175.1:47719 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:26.777357 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:26.772200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:53205)
1026 21:07:26.772504 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:26.772517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:26.772643 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:26.772932 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:26.772935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:26.773152 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:26.773357 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.773364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.774109 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.774114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:26.774502 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.774513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.774622 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.775325 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:26.775339 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:26.777196 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
W20251026 21:07:26.777460 13607 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:26.772273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:53205 (local address 127.3.175.62:43813)
1026 21:07:26.772427 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:26.772432 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:26.772590 (+ 158us) server_negotiation.cc:408] Connection header received
1026 21:07:26.772739 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:26.772743 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:26.772794 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:26.772878 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:26.773488 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.773992 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.774616 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.774829 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.777343 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:53205: BlockingRecv error: recv got EOF from 127.3.175.2:53205 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:26 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:26.959226 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:26.954425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:51441)
1026 21:07:26.954687 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:26.954701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:26.954791 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:26.955017 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:26.955019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:26.955218 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:26.955474 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.955482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.956153 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.956155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:26.956525 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:26.956531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.956622 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.957253 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:26.957268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:26.959059 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20251026 21:07:26.959347 13607 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:26.954603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:51441 (local address 127.3.175.62:43813)
1026 21:07:26.954726 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:26.954729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:26.954743 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:26.954871 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:26.954873 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:26.954920 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:26.954993 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:26.955580 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.956062 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:26.956613 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:26.956732 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:26.959202 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:51441: BlockingRecv error: recv got EOF from 127.3.175.3:51441 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:27.595432 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:27.590079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:54095)
1026 21:07:27.590373 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:27.590392 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:27.590498 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:27.590753 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:27.590756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:27.590954 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:27.591186 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.591193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.591975 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.591979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:27.592642 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.592657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.592813 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.593425 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:27.593439 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:27.595268 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
W20251026 21:07:27.595587 13611 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:27.590215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:54095 (local address 127.3.175.62:43813)
1026 21:07:27.590488 (+ 273us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:27.590491 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:27.590504 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:27.590585 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:27.590588 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:27.590635 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:27.590716 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:27.591310 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.591838 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.592778 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.592986 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.595462 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:54095: BlockingRecv error: recv got EOF from 127.3.175.1:54095 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:27.783615 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:27.778265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:52531)
1026 21:07:27.778594 (+ 329us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:27.778613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:27.778744 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:27.779040 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:27.779043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:27.779272 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:27.779539 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.779550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.780297 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.780300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:27.780689 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.780696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.780794 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.781467 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:27.781480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:27.783494 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":116,"threads_started":1}
W20251026 21:07:27.783792 13611 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:27.778382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:52531 (local address 127.3.175.62:43813)
1026 21:07:27.778517 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:27.778523 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:27.778702 (+ 179us) server_negotiation.cc:408] Connection header received
1026 21:07:27.778835 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:27.778838 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:27.778899 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:27.779000 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:27.779657 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.780178 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.780801 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.781119 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.783669 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:52531: BlockingRecv error: recv got EOF from 127.3.175.2:52531 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:27 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:27.965093 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:27.960083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:36933)
1026 21:07:27.960394 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:27.960408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:27.960532 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:27.960787 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:27.960790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:27.961006 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:27.961214 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.961221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.961981 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.961987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:27.962389 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:27.962397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.962513 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.963193 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:27.963207 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:27.964925 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
W20251026 21:07:27.965219 13611 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:27.960157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:36933 (local address 127.3.175.62:43813)
1026 21:07:27.960325 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:27.960330 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:27.960489 (+ 159us) server_negotiation.cc:408] Connection header received
1026 21:07:27.960625 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:27.960629 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:27.960683 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:27.960756 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:27.961343 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.961842 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:27.962506 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:27.962703 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:27.965113 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:36933: BlockingRecv error: recv got EOF from 127.3.175.3:36933 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:28.605675 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:28.596478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:57405)
1026 21:07:28.596914 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:28.596934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:28.597058 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:28.597350 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:28.597353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:28.597563 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:28.597760 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.597767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.598572 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.598576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:28.598960 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.598967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.599112 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.603617 (+ 4505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:28.603639 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:28.605547 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
W20251026 21:07:28.605926 13615 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:28.596566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:57405 (local address 127.3.175.62:43813)
1026 21:07:28.596857 (+ 291us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:28.596862 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:28.597007 (+ 145us) server_negotiation.cc:408] Connection header received
1026 21:07:28.597151 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:28.597155 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:28.597217 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:28.597320 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:28.597927 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.598461 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.599072 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.599267 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.605780 (+ 6513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:57405: BlockingRecv error: recv got EOF from 127.3.175.1:57405 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:28.789932 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:28.784596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:46399)
1026 21:07:28.784920 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:28.784933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:28.785042 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:28.785425 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:28.785428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:28.785615 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:28.785809 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.785815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.786607 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.786609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:28.786986 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.786992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.787088 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.787737 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:28.787763 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:28.789809 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":77,"threads_started":1}
W20251026 21:07:28.790114 13615 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:28.784729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:46399 (local address 127.3.175.62:43813)
1026 21:07:28.784907 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:28.784911 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:28.785008 (+ 97us) server_negotiation.cc:408] Connection header received
1026 21:07:28.785212 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:28.785215 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:28.785290 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:28.785383 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:28.785973 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.786498 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.787084 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.787225 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.789970 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:46399: BlockingRecv error: recv got EOF from 127.3.175.2:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:28 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:28.971226 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:28.966224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:46251)
1026 21:07:28.966552 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:28.966570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:28.966709 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:28.967024 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:28.967027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:28.967252 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:28.967495 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.967502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.968229 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.968233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:28.968605 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:28.968612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.968724 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.969319 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:28.969332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:28.971104 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
W20251026 21:07:28.971482 13615 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:28.966368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:46251 (local address 127.3.175.62:43813)
1026 21:07:28.966555 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:28.966562 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:28.966660 (+ 98us) server_negotiation.cc:408] Connection header received
1026 21:07:28.966813 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:28.966816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:28.966867 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:28.966982 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:28.967622 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.968118 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:28.968724 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:28.968937 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:28.971351 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:46251: BlockingRecv error: recv got EOF from 127.3.175.3:46251 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:29.612571 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:29.606672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:37003)
1026 21:07:29.607003 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:29.607018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:29.607152 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:29.607527 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:29.607532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:29.607796 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:29.608143 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.608170 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.609015 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.609019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:29.609590 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.609604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.609726 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.610426 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:29.610443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:29.612454 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
W20251026 21:07:29.612731 13619 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:29.606801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:37003 (local address 127.3.175.62:43813)
1026 21:07:29.607052 (+ 251us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:29.607056 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:29.607120 (+ 64us) server_negotiation.cc:408] Connection header received
1026 21:07:29.607266 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:29.607270 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:29.607355 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:29.607454 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:29.608323 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.608887 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.609714 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.609925 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.612584 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:37003: BlockingRecv error: recv got EOF from 127.3.175.1:37003 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:29.795817 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:29.790718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:57083)
1026 21:07:29.791022 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:29.791041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:29.791177 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:29.791480 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:29.791484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:29.791706 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:29.791896 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.791902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.792659 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.792663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:29.793054 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.793062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.793194 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.793829 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:29.793843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:29.795637 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20251026 21:07:29.795959 13619 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:29.790902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:57083 (local address 127.3.175.62:43813)
1026 21:07:29.791078 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:29.791085 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:29.791101 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:07:29.791314 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:29.791316 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:29.791364 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:29.791437 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:29.792037 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.792542 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.793182 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.793387 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.795809 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:57083: BlockingRecv error: recv got EOF from 127.3.175.2:57083 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:29 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:29.977443 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:29.972294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:36665)
1026 21:07:29.972601 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:29.972615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:29.972720 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:29.972988 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:29.972990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:29.973212 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:29.973411 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.973418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.974179 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.974183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:29.974608 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:29.974616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.974738 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.975436 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:29.975450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:29.977324 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
W20251026 21:07:29.977573 13619 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:29.972389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:36665 (local address 127.3.175.62:43813)
1026 21:07:29.972544 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:29.972550 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:29.972684 (+ 134us) server_negotiation.cc:408] Connection header received
1026 21:07:29.972797 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:29.972800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:29.972851 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:29.972963 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:29.973537 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.974040 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:29.974722 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:29.974849 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:29.977464 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:36665: BlockingRecv error: recv got EOF from 127.3.175.3:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:30.618674 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:30.613390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:60901)
1026 21:07:30.613694 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:30.613710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:30.613894 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:30.614257 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:30.614260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:30.614476 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:30.614682 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.614688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.615543 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.615547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:30.615960 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.615968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.616076 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.616690 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:30.616705 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:30.618552 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":116,"threads_started":1}
W20251026 21:07:30.619009 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:30.613460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:60901 (local address 127.3.175.62:43813)
1026 21:07:30.613717 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:30.613720 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:30.613795 (+ 75us) server_negotiation.cc:408] Connection header received
1026 21:07:30.614003 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:30.614006 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:30.614088 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:30.614183 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:30.614816 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.615381 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.616088 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.616235 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.618727 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:60901: BlockingRecv error: recv got EOF from 127.3.175.1:60901 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:30.801779 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:30.796781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:50159)
1026 21:07:30.797123 (+ 342us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:30.797169 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:30.797293 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:30.797549 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:30.797552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:30.797755 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:30.797958 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.797964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.798690 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.798693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:30.799064 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.799071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.799187 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.799800 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:30.799813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:30.801611 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
W20251026 21:07:30.801860 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:30.796946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:50159 (local address 127.3.175.62:43813)
1026 21:07:30.797117 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:30.797122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:30.797250 (+ 128us) server_negotiation.cc:408] Connection header received
1026 21:07:30.797385 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:30.797389 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:30.797440 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:30.797512 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:30.798083 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.798577 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.799213 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.799469 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.801746 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:50159: BlockingRecv error: recv got EOF from 127.3.175.2:50159 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:30 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:30.983899 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:30.978418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:40251)
1026 21:07:30.978733 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:30.978746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:30.978858 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:30.979181 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:30.979185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:30.979414 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:30.979627 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.979634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.980367 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.980370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:30.980770 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:30.980778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.980944 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.981592 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:30.981612 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:30.983737 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
W20251026 21:07:30.984016 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:30.978537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:40251 (local address 127.3.175.62:43813)
1026 21:07:30.978677 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:30.978681 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:30.978819 (+ 138us) server_negotiation.cc:408] Connection header received
1026 21:07:30.978950 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:30.978953 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:30.979004 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:30.979087 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:30.979747 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.980245 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:30.980951 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:30.981079 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:30.983879 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:40251: BlockingRecv error: recv got EOF from 127.3.175.3:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:31.625288 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:31.619547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:48453)
1026 21:07:31.619854 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:31.619875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:31.619978 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:31.620278 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:31.620282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:31.620467 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:31.620667 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.620673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.621554 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.621558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:31.621999 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.622008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.622121 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.622722 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:31.622737 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:31.625115 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
W20251026 21:07:31.625376 13627 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:31.619651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:48453 (local address 127.3.175.62:43813)
1026 21:07:31.619882 (+ 231us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:31.619903 (+ 21us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:31.619922 (+ 19us) server_negotiation.cc:408] Connection header received
1026 21:07:31.620066 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:31.620068 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:31.620116 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:31.620211 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:31.620774 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.621336 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.622122 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.622270 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.625253 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:48453: BlockingRecv error: recv got EOF from 127.3.175.1:48453 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:31.807830 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:31.802696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:42111)
1026 21:07:31.803085 (+ 389us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:31.803107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:31.803234 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:31.803552 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:31.803556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:31.803778 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:31.803987 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.803995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.804767 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.804770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:31.805169 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.805176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.805280 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.805853 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:31.805867 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:31.807715 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":93,"threads_started":1}
W20251026 21:07:31.808036 13627 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:31.802846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:42111 (local address 127.3.175.62:43813)
1026 21:07:31.803011 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:31.803016 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:31.803190 (+ 174us) server_negotiation.cc:408] Connection header received
1026 21:07:31.803367 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:31.803371 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:31.803424 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:31.803501 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:31.804137 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.804657 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.805290 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.805436 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.807906 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:42111: BlockingRecv error: recv got EOF from 127.3.175.2:42111 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:31 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:31.989545 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:31.984691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:39343)
1026 21:07:31.985005 (+ 314us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:31.985018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:31.985118 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:31.985352 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:31.985354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:31.985532 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:31.985726 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.985731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.986489 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.986493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:31.986936 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:31.986945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.987051 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.987632 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:31.987646 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:31.989424 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":76,"threads_started":1}
W20251026 21:07:31.989691 13627 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:31.984781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:39343 (local address 127.3.175.62:43813)
1026 21:07:31.984935 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:31.984938 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:31.985082 (+ 144us) server_negotiation.cc:408] Connection header received
1026 21:07:31.985190 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:31.985192 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:31.985239 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:31.985319 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:31.985838 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.986354 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:31.987036 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:31.987164 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:31.989551 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:39343: BlockingRecv error: recv got EOF from 127.3.175.3:39343 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:32.631134 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:32.626210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:35783)
1026 21:07:32.626507 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:32.626521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:32.626625 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:32.626912 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:32.626915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:32.627125 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:32.627391 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.627402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.628146 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.628149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:32.628532 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.628538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.628629 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.629241 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:32.629255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:32.631016 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
W20251026 21:07:32.631371 13631 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:32.626359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:35783 (local address 127.3.175.62:43813)
1026 21:07:32.626609 (+ 250us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:32.626615 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:32.626630 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:32.626715 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:32.626717 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:32.626774 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:32.626882 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:32.627513 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.628037 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.628617 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.628742 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.631189 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:35783: BlockingRecv error: recv got EOF from 127.3.175.1:35783 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:32.813426 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:32.808640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:50439)
1026 21:07:32.808949 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:32.808963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:32.809069 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:32.809306 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:32.809309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:32.809488 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:32.809683 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.809689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.810555 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.810558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:32.810936 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.810943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.811033 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.811598 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:32.811611 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:32.813308 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
W20251026 21:07:32.813563 13631 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:32.808720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:50439 (local address 127.3.175.62:43813)
1026 21:07:32.808878 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:32.808882 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:32.809029 (+ 147us) server_negotiation.cc:408] Connection header received
1026 21:07:32.809148 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:32.809151 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:32.809201 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:32.809272 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:32.809785 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.810463 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.811028 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.811157 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.813450 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:50439: BlockingRecv error: recv got EOF from 127.3.175.2:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:32 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:32.995813 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:32.990490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:49789)
1026 21:07:32.990806 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:32.990821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:32.990935 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:32.991248 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:32.991251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:32.991464 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:32.991665 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.991672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.992400 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.992448 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:32.992868 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:32.992878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.992990 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.993640 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:32.993654 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:32.995653 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
W20251026 21:07:32.995909 13631 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:32.990624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:49789 (local address 127.3.175.62:43813)
1026 21:07:32.990805 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:32.990809 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:32.990929 (+ 120us) server_negotiation.cc:408] Connection header received
1026 21:07:32.991030 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:32.991033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:32.991084 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:32.991169 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:32.991773 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.992282 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:32.993009 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:32.993211 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:32.995796 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:49789: BlockingRecv error: recv got EOF from 127.3.175.3:49789 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:33.641973 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:33.631996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:55361)
1026 21:07:33.632359 (+ 363us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:33.632375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:33.632492 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:33.632813 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:33.632816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:33.633060 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:33.633346 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.633354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.634560 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.634564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:33.635253 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.635267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.635411 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:33.639483 (+ 4072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:33.639499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:33.641853 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
W20251026 21:07:33.642071 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:33.632089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:55361 (local address 127.3.175.62:43813)
1026 21:07:33.632423 (+ 334us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:33.632430 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:33.632450 (+ 20us) server_negotiation.cc:408] Connection header received
1026 21:07:33.632607 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:33.632613 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:33.632681 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:33.632795 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:33.633485 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.634444 (+ 959us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.635437 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.635677 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:33.641967 (+ 6290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:55361: BlockingRecv error: recv got EOF from 127.3.175.1:55361 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":75,"threads_started":1}
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:33.819346 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:33.814185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:35655)
1026 21:07:33.814451 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:33.814465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:33.814578 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:33.814803 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:33.814805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:33.815006 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:33.815225 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.815232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.816019 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.816022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:33.816419 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.816427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.816519 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:33.817117 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:33.817131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:33.819167 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
W20251026 21:07:33.819475 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:33.814283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:35655 (local address 127.3.175.62:43813)
1026 21:07:33.814457 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:33.814461 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:33.814531 (+ 70us) server_negotiation.cc:408] Connection header received
1026 21:07:33.814655 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:33.814658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:33.814706 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:33.814776 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:33.815370 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.815901 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.816531 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.816680 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:33.819359 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:35655: BlockingRecv error: recv got EOF from 127.3.175.2:35655 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:33 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:34.002419 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:33.996844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:55881)
1026 21:07:33.997158 (+ 314us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:33.997172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:33.997283 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:33.997558 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:33.997560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:33.997758 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:33.998021 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.998029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.999020 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.999024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:33.999554 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:33.999564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.999680 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.000466 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:34.000480 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:34.002301 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
W20251026 21:07:34.002687 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:33.996998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:55881 (local address 127.3.175.62:43813)
1026 21:07:33.997155 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:33.997159 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:33.997240 (+ 81us) server_negotiation.cc:408] Connection header received
1026 21:07:33.997385 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:33.997388 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:33.997440 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:33.997505 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:33.998236 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.998784 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:33.999682 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:33.999909 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.002451 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:55881: BlockingRecv error: recv got EOF from 127.3.175.3:55881 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 26 21:07:34 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:34 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:34.647485 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:34.642805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:47627)
1026 21:07:34.643104 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:34.643117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:34.643217 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:34.643452 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:34.643454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:34.643642 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:34.643846 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:34.643852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.644568 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:34.644570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:34.644955 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:34.644961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.645065 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.645647 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:34.645660 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:34.647368 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
W20251026 21:07:34.647620 13639 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:34.642845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:47627 (local address 127.3.175.62:43813)
1026 21:07:34.643088 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:34.643091 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:34.643186 (+ 95us) server_negotiation.cc:408] Connection header received
1026 21:07:34.643305 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:34.643307 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:34.643352 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:34.643428 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:34.643946 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.644475 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:34.645062 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.645205 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.647502 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:47627: BlockingRecv error: recv got EOF from 127.3.175.1:47627 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
Oct 26 21:07:34 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:34 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:34.825397 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:34.820251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:46767)
1026 21:07:34.820672 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:34.820686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:34.820801 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:34.821090 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:34.821093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:34.821316 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:34.821509 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:34.821515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.822266 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:34.822270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:34.822647 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:34.822653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.822767 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.823463 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:34.823477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:34.825227 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
W20251026 21:07:34.825486 13639 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:34.820321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:46767 (local address 127.3.175.62:43813)
1026 21:07:34.820460 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:34.820464 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:34.820758 (+ 294us) server_negotiation.cc:408] Connection header received
1026 21:07:34.820915 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:34.820918 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:34.820968 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:34.821062 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:34.821640 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.822153 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:34.822755 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:34.822982 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:34.825369 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:46767: BlockingRecv error: recv got EOF from 127.3.175.2:46767 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:35.008242 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:35.003177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:39677)
1026 21:07:35.003485 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:35.003502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:35.003612 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:35.003940 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:35.003945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:35.004142 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:35.004332 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.004338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.005123 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.005129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:35.005523 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.005530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.005642 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.006290 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:35.006305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:35.008124 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":109,"threads_started":1}
W20251026 21:07:35.008419 13639 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:35.003376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:39677 (local address 127.3.175.62:43813)
1026 21:07:35.003505 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:35.003510 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:35.003560 (+ 50us) server_negotiation.cc:408] Connection header received
1026 21:07:35.003717 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:35.003722 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:35.003772 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:35.003846 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:35.004485 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.004996 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.005627 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.005825 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.008280 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:39677: BlockingRecv error: recv got EOF from 127.3.175.3:39677 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:35.653362 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:35.648331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:44547)
1026 21:07:35.648613 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:35.648626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:35.648729 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:35.649074 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:35.649077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:35.649284 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:35.649502 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.649510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.650244 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.650248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:35.650659 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.650666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.650775 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.651387 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:35.651402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:35.653242 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
W20251026 21:07:35.653538 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:35.648395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:44547 (local address 127.3.175.62:43813)
1026 21:07:35.648641 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:35.648644 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:35.648676 (+ 32us) server_negotiation.cc:408] Connection header received
1026 21:07:35.648805 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:35.648808 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:35.648855 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:35.648939 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:35.649616 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.650139 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.650752 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.650882 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.653396 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:44547: BlockingRecv error: recv got EOF from 127.3.175.1:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:35 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:35.831403 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:35.826332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:52593)
1026 21:07:35.826638 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:35.826652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:35.826761 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:35.827141 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:35.827144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:35.827380 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:35.827575 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.827581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.828363 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.828366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:35.828744 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:35.828750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.828837 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.829429 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:35.829443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:35.831229 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20251026 21:07:35.831526 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:35.826415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:52593 (local address 127.3.175.62:43813)
1026 21:07:35.826575 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:35.826583 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:35.826742 (+ 159us) server_negotiation.cc:408] Connection header received
1026 21:07:35.826964 (+ 222us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:35.826967 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:35.827022 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:35.827112 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:35.827724 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.828260 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:35.828914 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:35.829074 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:35.831415 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:52593: BlockingRecv error: recv got EOF from 127.3.175.2:52593 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:36.014417 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:36.009179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:38855)
1026 21:07:36.009481 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:36.009496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:36.009596 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:36.009866 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:36.009869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:36.010071 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:36.010267 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.010273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.010954 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.010957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:36.011377 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.011390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.011485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.012152 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:36.012168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:36.014296 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
W20251026 21:07:36.014547 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:36.009397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:38855 (local address 127.3.175.62:43813)
1026 21:07:36.009536 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:36.009540 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:36.009554 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:36.009686 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:36.009689 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:36.009737 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:36.009823 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:36.010371 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.010854 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.011502 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.011670 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.014420 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:38855: BlockingRecv error: recv got EOF from 127.3.175.3:38855 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:36.659663 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:36.654375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:60843)
1026 21:07:36.654679 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:36.654692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:36.654793 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:36.655106 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:36.655108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:36.655340 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:36.655539 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.655545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.656390 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.656393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:36.656764 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.656770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.656862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.657408 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:36.657420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:36.659493 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
W20251026 21:07:36.659755 13647 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:36.654486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:60843 (local address 127.3.175.62:43813)
1026 21:07:36.654744 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:36.654747 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:36.654764 (+ 17us) server_negotiation.cc:408] Connection header received
1026 21:07:36.654886 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:36.654888 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:36.654976 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:36.655070 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:36.655678 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.656284 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.656858 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.656998 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.659629 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:60843: BlockingRecv error: recv got EOF from 127.3.175.1:60843 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:36 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:36.837507 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:36.832389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:48459)
1026 21:07:36.832662 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:36.832677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:36.832782 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:36.833010 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:36.833012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:36.833215 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:36.833428 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.833434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.834204 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.834207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:36.834581 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:36.834587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.834680 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.835236 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:36.835248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:36.837335 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
W20251026 21:07:36.837561 13647 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:36.832475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:48459 (local address 127.3.175.62:43813)
1026 21:07:36.832619 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:36.832623 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:36.832743 (+ 120us) server_negotiation.cc:408] Connection header received
1026 21:07:36.832859 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:36.832862 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:36.832913 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:36.832987 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:36.833528 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.834067 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:36.834675 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:36.834808 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:36.837458 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:48459: BlockingRecv error: recv got EOF from 127.3.175.2:48459 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:37.020368 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:37.015341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:35179)
1026 21:07:37.015620 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:37.015634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:37.015756 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:37.016060 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:37.016063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:37.016275 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:37.016466 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.016473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.017199 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.017202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:37.017580 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.017586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.017695 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.018268 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:37.018282 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:37.020250 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20251026 21:07:37.020495 13647 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:37.015411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:35179 (local address 127.3.175.62:43813)
1026 21:07:37.015556 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:37.015562 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:37.015706 (+ 144us) server_negotiation.cc:408] Connection header received
1026 21:07:37.015861 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:37.015864 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:37.015953 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:37.016023 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:37.016586 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.017090 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.017684 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.017894 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.020390 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:35179: BlockingRecv error: recv got EOF from 127.3.175.3:35179 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:37.665318 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:37.660462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:58659)
1026 21:07:37.660749 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:37.660763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:37.660861 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:37.661108 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:37.661111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:37.661302 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:37.661496 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.661501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.662193 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.662195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:37.662572 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.662578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.662661 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.663277 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:37.663327 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:37.665155 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20251026 21:07:37.665465 13651 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:37.660545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:58659 (local address 127.3.175.62:43813)
1026 21:07:37.660804 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:37.660807 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:37.660822 (+ 15us) server_negotiation.cc:408] Connection header received
1026 21:07:37.660948 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:37.660950 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:37.660998 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:37.661077 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:37.661587 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.662094 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.662646 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.662787 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.665336 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:58659: BlockingRecv error: recv got EOF from 127.3.175.1:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:37 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:37.843137 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:37.838407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:57495)
1026 21:07:37.838661 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:37.838675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:37.838768 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:37.839022 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:37.839024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:37.839221 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:37.839450 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.839457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.840167 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.840170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:37.840541 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:37.840547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.840638 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.841188 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:37.841202 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:37.842966 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20251026 21:07:37.843201 13651 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:37.838688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:57495 (local address 127.3.175.62:43813)
1026 21:07:37.838819 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:37.838823 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:37.838835 (+ 12us) server_negotiation.cc:408] Connection header received
1026 21:07:37.838871 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:37.838874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:37.838923 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:37.839000 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:37.839555 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.840063 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:37.840625 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:37.840748 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:37.843090 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:57495: BlockingRecv error: recv got EOF from 127.3.175.2:57495 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:38.026525 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:38.021340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:48051)
1026 21:07:38.021629 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:38.021642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:38.021753 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:38.022036 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:38.022038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:38.022226 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:38.022423 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.022430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.023270 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.023274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:38.023801 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.023813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.023948 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.024563 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:38.024577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:38.026408 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
W20251026 21:07:38.026646 13651 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:38.021429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:48051 (local address 127.3.175.62:43813)
1026 21:07:38.021566 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:38.021570 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:38.021710 (+ 140us) server_negotiation.cc:408] Connection header received
1026 21:07:38.021856 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:38.021858 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:38.021931 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:38.021999 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:38.022535 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.023150 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.023944 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.024126 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.026541 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:48051: BlockingRecv error: recv got EOF from 127.3.175.3:48051 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:38.675325 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:38.666221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:36695)
1026 21:07:38.666526 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:38.666540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:38.666642 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:38.666878 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:38.666880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:38.667074 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:38.667268 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.667274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.668022 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.668025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:38.668399 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.668405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.668491 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.673247 (+ 4756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:38.673264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:38.675126 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20251026 21:07:38.675451 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:38.666318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:36695 (local address 127.3.175.62:43813)
1026 21:07:38.666637 (+ 319us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:38.666641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:38.666657 (+ 16us) server_negotiation.cc:408] Connection header received
1026 21:07:38.666724 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:38.666727 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:38.666775 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:38.666847 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:38.667405 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.667930 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.668492 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.668627 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.675321 (+ 6694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:36695: BlockingRecv error: recv got EOF from 127.3.175.1:36695 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:38 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:38.849536 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:38.844131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:56273)
1026 21:07:38.844454 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:38.844467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:38.844591 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:38.844859 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:38.844862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:38.845063 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:38.845266 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.845272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.846021 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.846027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:38.846418 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:38.846425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.846538 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.847221 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:38.847234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:38.849423 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20251026 21:07:38.849651 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:38.844217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:56273 (local address 127.3.175.62:43813)
1026 21:07:38.844397 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:38.844402 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:38.844547 (+ 145us) server_negotiation.cc:408] Connection header received
1026 21:07:38.844691 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:38.844695 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:38.844746 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:38.844817 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:38.845399 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.845890 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:38.846535 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:38.846751 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:38.849552 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:56273: BlockingRecv error: recv got EOF from 127.3.175.2:56273 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.3@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:39.032753 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:39.027450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.3:34705)
1026 21:07:39.027887 (+ 437us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:39.027908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:39.028023 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:39.028314 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:39.028318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:39.028552 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:39.028825 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.028834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.029592 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.029595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:39.029995 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.030003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.030109 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.030686 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:39.030700 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:39.032590 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":81,"threads_started":1}
W20251026 21:07:39.032850 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:39.027615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:34705 (local address 127.3.175.62:43813)
1026 21:07:39.027806 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:39.027811 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:39.027979 (+ 168us) server_negotiation.cc:408] Connection header received
1026 21:07:39.028116 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:39.028119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:39.028178 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:39.028264 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:39.028956 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.029497 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.030087 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.030223 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.032728 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.3:34705: BlockingRecv error: recv got EOF from 127.3.175.3:34705 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.1@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:39.681610 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:39.676221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.1:47475)
1026 21:07:39.676510 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:39.676523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:39.676607 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:39.676939 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:39.676942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:39.677126 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:39.677322 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.677328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.678107 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.678109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:39.678486 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.678492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.678583 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.679222 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:39.679235 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:39.681485 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
W20251026 21:07:39.681820 13659 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:39.676437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:47475 (local address 127.3.175.62:43813)
1026 21:07:39.676694 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:39.676698 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:39.676711 (+ 13us) server_negotiation.cc:408] Connection header received
1026 21:07:39.676749 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:39.676751 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:39.676796 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:39.676899 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:39.677451 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.678001 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.678585 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.678724 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.681666 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.1:47475: BlockingRecv error: recv got EOF from 127.3.175.1:47475 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
Oct 26 21:07:39 dist-test-slave-th6x krb5kdc[12592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.175.2@KRBTEST.COM for oryx/127.3.175.62@KRBTEST.COM, Server not found in Kerberos database
I20251026 21:07:39.856020 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:39.850624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43813 (local address 127.3.175.2:50859)
1026 21:07:39.850959 (+ 335us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:39.850973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:39.851081 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:39.851387 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:39.851390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:39.851617 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:39.851899 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.851908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.852883 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.852886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:39.853282 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:39.853289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.853395 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.853960 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:39.853974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:39.855901 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.175.62:43813: Server oryx/127.3.175.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
W20251026 21:07:39.856159 13659 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:39.850738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:50859 (local address 127.3.175.62:43813)
1026 21:07:39.850923 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:39.850929 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:39.851059 (+ 130us) server_negotiation.cc:408] Connection header received
1026 21:07:39.851183 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:39.851185 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:39.851238 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:39.851370 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:39.852034 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.852725 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:39.853392 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:39.853527 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:39.856052 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.175.2:50859: BlockingRecv error: recv got EOF from 127.3.175.2:50859 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 21:07:39.934948 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13184
I20251026 21:07:39.940872 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13336
I20251026 21:07:39.946861 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13468
I20251026 21:07:39.952672 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13114
2025-10-26T21:07:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18970 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-26T21:07:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:39Z Disabled control of system clock
I20251026 21:07:39.984169 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:39273
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251026 21:07:40.096026 13664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:40.096282 13664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:40.096328 13664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:40.099649 13664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:40.099722 13664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:40.099738 13664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:40.099757 13664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:40.099774 13664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:40.103782 13664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:39273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:40.104941 13664 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:40.106060 13664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:40.112088 13669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.112231 13670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.112285 13664 server_base.cc:1047] running on GCE node
W20251026 21:07:40.112088 13672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.113092 13664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:40.114091 13664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:40.115245 13664 hybrid_clock.cc:648] HybridClock initialized: now 1761512860115235 us; error 28 us; skew 500 ppm
I20251026 21:07:40.117177 13664 webserver.cc:492] Webserver started at http://127.3.175.62:45879/ using document root <none> and password file <none>
I20251026 21:07:40.117786 13664 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:40.117843 13664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:40.118060 13664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:40.119971 13664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "c549e95f0243404f91a2183d277e6ca4"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "510fe7f4497b93c55690ea2b14d16311"
server_key_iv: "352e92951676be53d015b8f395354e73"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.120472 13664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "c549e95f0243404f91a2183d277e6ca4"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "510fe7f4497b93c55690ea2b14d16311"
server_key_iv: "352e92951676be53d015b8f395354e73"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.124106 13664 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:40.126734 13678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.128060 13664 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:40.128209 13664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "c549e95f0243404f91a2183d277e6ca4"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "510fe7f4497b93c55690ea2b14d16311"
server_key_iv: "352e92951676be53d015b8f395354e73"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.128314 13664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:40.156276 13664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:40.157008 13664 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:40.157207 13664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:40.164738 13730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:39273 every 8 connection(s)
I20251026 21:07:40.164741 13664 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:39273
I20251026 21:07:40.165939 13664 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:40.168973 13731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:40.170303 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13664
I20251026 21:07:40.170444 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:40.170773 3772 external_mini_cluster.cc:1468] Setting key 7b25cdde6351b9ef7cbac0013efb493b
I20251026 21:07:40.175743 13731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4: Bootstrap starting.
I20251026 21:07:40.178150 13731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:40.178740 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.172287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.3.175.62:39273)
1026 21:07:40.173064 (+ 777us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:40.173073 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:40.173108 (+ 35us) server_negotiation.cc:408] Connection header received
1026 21:07:40.173606 (+ 498us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:40.173622 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:40.173978 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:40.174299 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:40.174798 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.175596 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.176301 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.176585 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.177169 (+ 584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:40.177212 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:40.177221 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:40.177580 (+ 359us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:40.177620 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:40.177633 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:40.177731 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:40.177885 (+ 154us) server_negotiation.cc:300] Negotiation successful
1026 21:07:40.178130 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":182,"threads_started":1}
I20251026 21:07:40.178905 13731 log.cc:826] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:40.180881 13731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4: No bootstrap required, opened a new log
I20251026 21:07:40.183496 13731 raft_consensus.cc:359] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } }
I20251026 21:07:40.183763 13731 raft_consensus.cc:385] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:40.183940 13731 raft_consensus.cc:740] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c549e95f0243404f91a2183d277e6ca4, State: Initialized, Role: FOLLOWER
I20251026 21:07:40.184417 13731 consensus_queue.cc:260] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } }
I20251026 21:07:40.184563 13731 raft_consensus.cc:399] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:40.184635 13731 raft_consensus.cc:493] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:40.184741 13731 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:40.185688 13731 raft_consensus.cc:515] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } }
I20251026 21:07:40.186057 13731 leader_election.cc:304] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c549e95f0243404f91a2183d277e6ca4; no voters:
I20251026 21:07:40.186386 13731 leader_election.cc:290] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:40.186511 13736 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:40.186741 13736 raft_consensus.cc:697] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [term 1 LEADER]: Becoming Leader. State: Replica: c549e95f0243404f91a2183d277e6ca4, State: Running, Role: LEADER
I20251026 21:07:40.187187 13736 consensus_queue.cc:237] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } }
I20251026 21:07:40.187621 13731 sys_catalog.cc:565] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:40.188819 13737 sys_catalog.cc:455] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c549e95f0243404f91a2183d277e6ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } } }
I20251026 21:07:40.188974 13737 sys_catalog.cc:458] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:40.189355 13738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c549e95f0243404f91a2183d277e6ca4. Latest consensus state: current_term: 1 leader_uuid: "c549e95f0243404f91a2183d277e6ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c549e95f0243404f91a2183d277e6ca4" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 39273 } } }
I20251026 21:07:40.189396 13745 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:40.189425 13738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:40.192598 13745 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:40.198886 13745 catalog_manager.cc:1357] Generated new cluster ID: bbd8a067b7614ec7a37dcf02596481c8
I20251026 21:07:40.199000 13745 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:40.213438 13745 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:40.214808 13745 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:40.230598 13745 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c549e95f0243404f91a2183d277e6ca4: Generated new TSK 0
I20251026 21:07:40.231664 13745 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:07:40.237208 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--builtin_ntp_servers=127.3.175.20:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251026 21:07:40.355161 13755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:40.355475 13755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:40.355569 13755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:40.358997 13755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:40.359076 13755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:40.359194 13755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:40.363436 13755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:40.364691 13755 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:40.365907 13755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:40.373226 13760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.373258 13763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.373370 13761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.373731 13755 server_base.cc:1047] running on GCE node
I20251026 21:07:40.374166 13755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:40.374751 13755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:40.375917 13755 hybrid_clock.cc:648] HybridClock initialized: now 1761512860375879 us; error 50 us; skew 500 ppm
I20251026 21:07:40.377921 13755 webserver.cc:492] Webserver started at http://127.3.175.1:44881/ using document root <none> and password file <none>
I20251026 21:07:40.378520 13755 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:40.378584 13755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:40.378805 13755 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:40.380590 13755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "cb9348fe30ba4874ac5bb47ffe187edc"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "db5a5f60941b5a8b0cca3aef92218cd8"
server_key_iv: "630cbd82b8a542f007d2a893fca0b4c2"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.381078 13755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "cb9348fe30ba4874ac5bb47ffe187edc"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "db5a5f60941b5a8b0cca3aef92218cd8"
server_key_iv: "630cbd82b8a542f007d2a893fca0b4c2"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.384680 13755 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:40.387038 13769 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.388335 13755 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:40.388471 13755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "cb9348fe30ba4874ac5bb47ffe187edc"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "db5a5f60941b5a8b0cca3aef92218cd8"
server_key_iv: "630cbd82b8a542f007d2a893fca0b4c2"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.388581 13755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:40.412642 13755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:40.413439 13755 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:40.413655 13755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:40.414290 13755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:40.415407 13755 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:40.415468 13755 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.415541 13755 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:40.415577 13755 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.425906 13755 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:45001
I20251026 21:07:40.425988 13882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:45001 every 8 connection(s)
I20251026 21:07:40.427155 13755 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:40.433727 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13755
I20251026 21:07:40.433881 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:40.434219 3772 external_mini_cluster.cc:1468] Setting key f170754abe3170a126e010c5b80ba6f2
I20251026 21:07:40.436043 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.429268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:33251 (local address 127.3.175.62:39273)
1026 21:07:40.429454 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:40.429459 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:40.430463 (+ 1004us) server_negotiation.cc:408] Connection header received
1026 21:07:40.431311 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:40.431314 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:40.431368 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:40.431445 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:40.432503 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.433100 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.433792 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.434010 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.435488 (+ 1478us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:40.435509 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:40.435512 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:40.435528 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:40.435556 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:40.435565 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:40.435647 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:40.435833 (+ 186us) server_negotiation.cc:300] Negotiation successful
1026 21:07:40.435905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251026 21:07:40.436395 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--builtin_ntp_servers=127.3.175.20:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251026 21:07:40.436831 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.429624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:39273 (local address 127.3.175.1:33251)
1026 21:07:40.430311 (+ 687us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:40.430348 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:40.431112 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:40.431609 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:40.431619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:40.431703 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:40.432344 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.432357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.433216 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.433219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:40.433654 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.433663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.433870 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.435211 (+ 1341us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:40.435213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:40.435253 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:40.435353 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:40.435663 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:40.435670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:40.435674 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 21:07:40.435825 (+ 151us) client_negotiation.cc:240] Negotiation successful
1026 21:07:40.436125 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20251026 21:07:40.438115 13883 heartbeater.cc:344] Connected to a master server at 127.3.175.62:39273
I20251026 21:07:40.438414 13883 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:40.439010 13883 heartbeater.cc:507] Master 127.3.175.62:39273 requested a full tablet report, sending...
I20251026 21:07:40.440768 13695 ts_manager.cc:194] Registered new tserver with Master: cb9348fe30ba4874ac5bb47ffe187edc (127.3.175.1:45001)
I20251026 21:07:40.442637 13695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.175.1:33251
W20251026 21:07:40.547891 13886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:40.548154 13886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:40.548206 13886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:40.552222 13886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:40.552296 13886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:40.552374 13886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:40.556470 13886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:40.557627 13886 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:40.558749 13886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:40.565562 13894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.565578 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.565681 13886 server_base.cc:1047] running on GCE node
W20251026 21:07:40.565547 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.566165 13886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:40.566708 13886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:40.567862 13886 hybrid_clock.cc:648] HybridClock initialized: now 1761512860567833 us; error 43 us; skew 500 ppm
I20251026 21:07:40.569831 13886 webserver.cc:492] Webserver started at http://127.3.175.2:36155/ using document root <none> and password file <none>
I20251026 21:07:40.570434 13886 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:40.570497 13886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:40.570689 13886 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:40.572584 13886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "115f68cc8ba34043849977cad159994b"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "20261be3e9154bfc8a83ff54b5b1cacc"
server_key_iv: "9b083af2c9f29bbf054f0c5c834ec6b8"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.573077 13886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "115f68cc8ba34043849977cad159994b"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "20261be3e9154bfc8a83ff54b5b1cacc"
server_key_iv: "9b083af2c9f29bbf054f0c5c834ec6b8"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.576614 13886 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:40.579058 13900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.580155 13886 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:40.580260 13886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "115f68cc8ba34043849977cad159994b"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "20261be3e9154bfc8a83ff54b5b1cacc"
server_key_iv: "9b083af2c9f29bbf054f0c5c834ec6b8"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.580374 13886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:40.607589 13886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:40.608317 13886 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:40.608506 13886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:40.609100 13886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:40.610118 13886 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:40.610188 13886 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.610242 13886 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:40.610275 13886 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.619921 13886 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:34597
I20251026 21:07:40.619984 14013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:34597 every 8 connection(s)
I20251026 21:07:40.620975 13886 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:40.622923 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 13886
I20251026 21:07:40.623059 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:40.623397 3772 external_mini_cluster.cc:1468] Setting key 0a0c31c9c33f61d6a0a9d57e9f9be0e6
I20251026 21:07:40.625587 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--builtin_ntp_servers=127.3.175.20:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251026 21:07:40.630012 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.622836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:46621 (local address 127.3.175.62:39273)
1026 21:07:40.623018 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:40.623023 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:40.624140 (+ 1117us) server_negotiation.cc:408] Connection header received
1026 21:07:40.625130 (+ 990us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:40.625134 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:40.625195 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:40.625286 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:40.626373 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.627106 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.627816 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.628005 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.629465 (+ 1460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:40.629488 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:40.629490 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:40.629504 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:40.629525 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:40.629532 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:40.629609 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:40.629780 (+ 171us) server_negotiation.cc:300] Negotiation successful
1026 21:07:40.629861 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251026 21:07:40.630743 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.623226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:39273 (local address 127.3.175.2:46621)
1026 21:07:40.623992 (+ 766us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:40.624035 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:40.624936 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:40.625431 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:40.625439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:40.625520 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:40.626213 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.626233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.627244 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.627247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:40.627700 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.627708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.627951 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.629224 (+ 1273us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:40.629226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:40.629261 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:40.629321 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:40.629622 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:40.629635 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:40.629640 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 21:07:40.629782 (+ 142us) client_negotiation.cc:240] Negotiation successful
1026 21:07:40.630051 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20251026 21:07:40.632234 14014 heartbeater.cc:344] Connected to a master server at 127.3.175.62:39273
I20251026 21:07:40.632508 14014 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:40.633154 14014 heartbeater.cc:507] Master 127.3.175.62:39273 requested a full tablet report, sending...
I20251026 21:07:40.634341 13695 ts_manager.cc:194] Registered new tserver with Master: 115f68cc8ba34043849977cad159994b (127.3.175.2:34597)
I20251026 21:07:40.635162 13695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.175.2:46621
W20251026 21:07:40.737766 14017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:40.738010 14017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:40.738059 14017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:40.741364 14017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:40.741428 14017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:40.741501 14017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:40.745395 14017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:39273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:40.746506 14017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:40.747653 14017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:40.754186 14023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.754192 14022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:40.754317 14025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:40.754475 14017 server_base.cc:1047] running on GCE node
I20251026 21:07:40.754842 14017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:40.755417 14017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:40.756578 14017 hybrid_clock.cc:648] HybridClock initialized: now 1761512860756565 us; error 26 us; skew 500 ppm
I20251026 21:07:40.758423 14017 webserver.cc:492] Webserver started at http://127.3.175.3:36589/ using document root <none> and password file <none>
I20251026 21:07:40.758991 14017 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:40.759049 14017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:40.759255 14017 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:40.760996 14017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "7d1bdd9f775b4af4ab33fa9ef09fef38"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "ba028b83bfcf7ac83c8fb14f38314e64"
server_key_iv: "ead600a8e879d463373d33776fa4848e"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.761445 14017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "7d1bdd9f775b4af4ab33fa9ef09fef38"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "ba028b83bfcf7ac83c8fb14f38314e64"
server_key_iv: "ead600a8e879d463373d33776fa4848e"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.764958 14017 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251026 21:07:40.767051 14031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.768088 14017 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251026 21:07:40.768200 14017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "7d1bdd9f775b4af4ab33fa9ef09fef38"
format_stamp: "Formatted at 2025-10-26 21:07:40 on dist-test-slave-th6x"
server_key: "ba028b83bfcf7ac83c8fb14f38314e64"
server_key_iv: "ead600a8e879d463373d33776fa4848e"
server_key_version: "encryptionkey@0"
I20251026 21:07:40.768299 14017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:40.793677 14017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:40.794458 14017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:40.794667 14017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:40.795333 14017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:40.796447 14017 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:40.796506 14017 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.796572 14017 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:40.796609 14017 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:40.806933 14017 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:38361
I20251026 21:07:40.806951 14144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:38361 every 8 connection(s)
I20251026 21:07:40.808059 14017 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:40.811852 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14017
I20251026 21:07:40.811970 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:40.812258 3772 external_mini_cluster.cc:1468] Setting key 9028a1a995e550e216a59b65121b644e
I20251026 21:07:40.816720 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.809966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:59609 (local address 127.3.175.62:39273)
1026 21:07:40.810110 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:40.810114 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:40.811080 (+ 966us) server_negotiation.cc:408] Connection header received
1026 21:07:40.812144 (+ 1064us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:40.812148 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:40.812201 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:40.812268 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:40.813480 (+ 1212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.814164 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.814827 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.814981 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.816303 (+ 1322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:40.816317 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:40.816320 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:40.816331 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:40.816348 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:40.816355 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:40.816418 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:40.816555 (+ 137us) server_negotiation.cc:300] Negotiation successful
1026 21:07:40.816607 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:07:40.817409 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:40.810278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:39273 (local address 127.3.175.3:59609)
1026 21:07:40.810936 (+ 658us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:40.810971 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:40.811878 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:40.812486 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:40.812497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:40.812598 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:40.813294 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.813309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.814296 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:40.814301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:40.814714 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:40.814722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:40.814915 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:40.816088 (+ 1173us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:40.816090 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:40.816123 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:40.816181 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:40.816427 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:40.816432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:40.816435 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 21:07:40.816546 (+ 111us) client_negotiation.cc:240] Negotiation successful
1026 21:07:40.816743 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20251026 21:07:40.818385 14145 heartbeater.cc:344] Connected to a master server at 127.3.175.62:39273
I20251026 21:07:40.818626 14145 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:40.819182 14145 heartbeater.cc:507] Master 127.3.175.62:39273 requested a full tablet report, sending...
I20251026 21:07:40.820194 13695 ts_manager.cc:194] Registered new tserver with Master: 7d1bdd9f775b4af4ab33fa9ef09fef38 (127.3.175.3:38361)
I20251026 21:07:40.820730 13695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.175.3:59609
I20251026 21:07:40.825990 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 21:07:40.828758 13734 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:40.827852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.3.175.62:39273)
1026 21:07:40.828014 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:40.828019 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:40.828139 (+ 120us) server_negotiation.cc:408] Connection header received
1026 21:07:40.828280 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:40.828284 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:40.828334 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:40.828403 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:40.828635 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38320: BlockingRecv error: recv got EOF from 127.0.0.1:38320 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20251026 21:07:40.828898 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13755
I20251026 21:07:40.834362 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13886
I20251026 21:07:40.840044 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14017
I20251026 21:07:40.845512 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 13664
2025-10-26T21:07:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (891 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-26T21:07:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:40Z Disabled control of system clock
I20251026 21:07:40.874888 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46747
--webserver_interface=127.3.175.62
--webserver_port=0
--builtin_ntp_servers=127.3.175.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:46747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 21:07:40.986995 14156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:40.987264 14156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:40.987372 14156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:40.990928 14156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:40.991005 14156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:40.991022 14156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:40.991041 14156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:40.991060 14156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:40.995121 14156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:46747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:46747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:40.996337 14156 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:40.997465 14156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:41.003464 14164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:41.003469 14161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:41.003464 14162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.003851 14156 server_base.cc:1047] running on GCE node
I20251026 21:07:41.004426 14156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:41.005462 14156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:41.006642 14156 hybrid_clock.cc:648] HybridClock initialized: now 1761512861006621 us; error 36 us; skew 500 ppm
I20251026 21:07:41.008677 14156 webserver.cc:492] Webserver started at http://127.3.175.62:42937/ using document root <none> and password file <none>
I20251026 21:07:41.009306 14156 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:41.009377 14156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:41.009574 14156 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:41.011435 14156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "7843a4867a774104851ee204610c66f8"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "acf041f07a84bd33f1b6dd616ff9795d"
server_key_iv: "b3c4ea2092ba49b96d701d6af0057c4a"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.011957 14156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "7843a4867a774104851ee204610c66f8"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "acf041f07a84bd33f1b6dd616ff9795d"
server_key_iv: "b3c4ea2092ba49b96d701d6af0057c4a"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.015661 14156 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:41.018218 14170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.019361 14156 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:41.019487 14156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "7843a4867a774104851ee204610c66f8"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "acf041f07a84bd33f1b6dd616ff9795d"
server_key_iv: "b3c4ea2092ba49b96d701d6af0057c4a"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.019593 14156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:41.024897 14156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:41.025564 14156 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:41.025746 14156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:41.033161 14156 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:46747
I20251026 21:07:41.033164 14222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:46747 every 8 connection(s)
I20251026 21:07:41.034279 14156 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:41.037269 14223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:41.040608 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14156
I20251026 21:07:41.040752 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:41.041030 3772 external_mini_cluster.cc:1468] Setting key 86da6bda50ae9719db9cf74b45d35377
I20251026 21:07:41.043761 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8: Bootstrap starting.
I20251026 21:07:41.046185 14223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:41.046661 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.042482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.3.175.62:46747)
1026 21:07:41.043175 (+ 693us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:41.043185 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:41.043221 (+ 36us) server_negotiation.cc:408] Connection header received
1026 21:07:41.043801 (+ 580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:41.043819 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:41.044193 (+ 374us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:41.044516 (+ 323us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:41.045402 (+ 886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:41.045407 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:41.045417 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:41.045428 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:41.045470 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:41.045481 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:41.045604 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:41.045728 (+ 124us) server_negotiation.cc:300] Negotiation successful
1026 21:07:41.045996 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20251026 21:07:41.046969 14223 log.cc:826] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:41.048988 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8: No bootstrap required, opened a new log
I20251026 21:07:41.051503 14223 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } }
I20251026 21:07:41.051695 14223 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:41.051738 14223 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7843a4867a774104851ee204610c66f8, State: Initialized, Role: FOLLOWER
I20251026 21:07:41.052153 14223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } }
I20251026 21:07:41.052307 14223 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:41.052385 14223 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:41.052486 14223 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:41.053459 14223 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } }
I20251026 21:07:41.053771 14223 leader_election.cc:304] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7843a4867a774104851ee204610c66f8; no voters:
I20251026 21:07:41.054037 14223 leader_election.cc:290] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:41.054141 14228 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:41.054353 14228 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7843a4867a774104851ee204610c66f8, State: Running, Role: LEADER
I20251026 21:07:41.054636 14228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } }
I20251026 21:07:41.055158 14223 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:41.056344 14229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7843a4867a774104851ee204610c66f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } } }
I20251026 21:07:41.056363 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7843a4867a774104851ee204610c66f8. Latest consensus state: current_term: 1 leader_uuid: "7843a4867a774104851ee204610c66f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7843a4867a774104851ee204610c66f8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 46747 } } }
I20251026 21:07:41.056631 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:41.056631 14229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:41.057588 14237 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:41.060447 14237 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:41.065799 14237 catalog_manager.cc:1357] Generated new cluster ID: 6dd4f8b68c5443109fe5cf95c88f3b04
I20251026 21:07:41.065964 14237 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:41.079339 14237 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:41.079659 14237 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:41.087329 14237 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7843a4867a774104851ee204610c66f8: Generated new TSK 0
I20251026 21:07:41.088110 14237 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 21:07:41.093691 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--builtin_ntp_servers=127.3.175.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 21:07:41.227197 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:41.227525 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:41.227596 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:41.231097 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:41.231173 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:41.231256 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:41.235422 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:41.236622 14247 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:41.237811 14247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:41.244838 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:41.244858 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.245158 14247 server_base.cc:1047] running on GCE node
W20251026 21:07:41.244830 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.245573 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:41.246194 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:41.247375 14247 hybrid_clock.cc:648] HybridClock initialized: now 1761512861247346 us; error 45 us; skew 500 ppm
I20251026 21:07:41.249388 14247 webserver.cc:492] Webserver started at http://127.3.175.1:46309/ using document root <none> and password file <none>
I20251026 21:07:41.249991 14247 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:41.250047 14247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:41.250242 14247 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:41.252123 14247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "1747acdb4c6c4ec58407a9829fd961cc"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "1f1887a019acc7c357efb8cab9f228a3"
server_key_iv: "b4123d8cc01ffbde016a832266459ccf"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.252601 14247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "1747acdb4c6c4ec58407a9829fd961cc"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "1f1887a019acc7c357efb8cab9f228a3"
server_key_iv: "b4123d8cc01ffbde016a832266459ccf"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.256328 14247 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:41.258827 14261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.259979 14247 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:41.260103 14247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "1747acdb4c6c4ec58407a9829fd961cc"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "1f1887a019acc7c357efb8cab9f228a3"
server_key_iv: "b4123d8cc01ffbde016a832266459ccf"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.260228 14247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:41.265295 14247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:41.265882 14247 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:41.266041 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:41.266530 14247 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:41.267563 14247 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:41.267614 14247 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.267654 14247 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:41.267675 14247 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.276520 14247 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:41961
I20251026 21:07:41.276564 14374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:41961 every 8 connection(s)
I20251026 21:07:41.277606 14247 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:41.280695 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14247
I20251026 21:07:41.280826 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:41.281152 3772 external_mini_cluster.cc:1468] Setting key 3532ad8a3386ede97dc592e093d80289
I20251026 21:07:41.283244 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--builtin_ntp_servers=127.3.175.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251026 21:07:41.283715 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.279459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:33969 (local address 127.3.175.62:46747)
1026 21:07:41.279610 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:41.279616 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:41.280733 (+ 1117us) server_negotiation.cc:408] Connection header received
1026 21:07:41.281779 (+ 1046us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:41.281783 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:41.281831 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:41.281923 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:41.283149 (+ 1226us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:41.283153 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:41.283156 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:41.283169 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:41.283191 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:41.283198 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:41.283278 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:41.283519 (+ 241us) server_negotiation.cc:300] Negotiation successful
1026 21:07:41.283567 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 21:07:41.284502 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.279817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46747 (local address 127.3.175.1:33969)
1026 21:07:41.280554 (+ 737us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:41.280605 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:41.281569 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:41.282098 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:41.282110 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:41.282240 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:41.282920 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:41.282922 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:41.282959 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:41.283014 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:41.283273 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:41.283314 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:41.283322 (+ 8us) client_negotiation.cc:769] Sending connection context
1026 21:07:41.283587 (+ 265us) client_negotiation.cc:240] Negotiation successful
1026 21:07:41.283838 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20251026 21:07:41.285848 14375 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46747
I20251026 21:07:41.286093 14375 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:41.286525 14375 heartbeater.cc:507] Master 127.3.175.62:46747 requested a full tablet report, sending...
I20251026 21:07:41.287998 14187 ts_manager.cc:194] Registered new tserver with Master: 1747acdb4c6c4ec58407a9829fd961cc (127.3.175.1:41961)
W20251026 21:07:41.395468 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:41.395721 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:41.395776 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:41.399176 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:41.399250 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:41.399386 14378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:41.403402 14378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:41.404621 14378 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:41.405793 14378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:41.412889 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:41.412983 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.412941 14378 server_base.cc:1047] running on GCE node
W20251026 21:07:41.413017 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.413581 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:41.414230 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:41.415350 14378 hybrid_clock.cc:648] HybridClock initialized: now 1761512861415339 us; error 36 us; skew 500 ppm
I20251026 21:07:41.417299 14378 webserver.cc:492] Webserver started at http://127.3.175.2:35659/ using document root <none> and password file <none>
I20251026 21:07:41.417886 14378 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:41.417948 14378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:41.418159 14378 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:41.419989 14378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "9cc250950a4344c0b37a0dc128dfc0f0"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "cda6dc0e656af156bf1578a8511b2b33"
server_key_iv: "e7a4b3642d5e545b199e9d8cdbceb59e"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.420506 14378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "9cc250950a4344c0b37a0dc128dfc0f0"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "cda6dc0e656af156bf1578a8511b2b33"
server_key_iv: "e7a4b3642d5e545b199e9d8cdbceb59e"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.424080 14378 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 21:07:41.426522 14392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.427592 14378 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:41.427698 14378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "9cc250950a4344c0b37a0dc128dfc0f0"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "cda6dc0e656af156bf1578a8511b2b33"
server_key_iv: "e7a4b3642d5e545b199e9d8cdbceb59e"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.427803 14378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:41.432741 14378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:41.433346 14378 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:41.433513 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:41.434051 14378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:41.435022 14378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:41.435070 14378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.435117 14378 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:41.435156 14378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.444306 14378 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:34475
I20251026 21:07:41.444341 14505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:34475 every 8 connection(s)
I20251026 21:07:41.445335 14378 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:41.449342 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14378
I20251026 21:07:41.449466 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:41.449733 3772 external_mini_cluster.cc:1468] Setting key e78cf6244f40db7c953f52827b310119
I20251026 21:07:41.451061 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.447186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:33931 (local address 127.3.175.62:46747)
1026 21:07:41.447371 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:41.447375 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:41.448388 (+ 1013us) server_negotiation.cc:408] Connection header received
1026 21:07:41.449224 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:41.449228 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:41.449280 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:41.449370 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:41.450595 (+ 1225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:41.450599 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:41.450601 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:41.450613 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:41.450633 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:41.450641 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:41.450724 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:41.450872 (+ 148us) server_negotiation.cc:300] Negotiation successful
1026 21:07:41.450921 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:07:41.451838 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--builtin_ntp_servers=127.3.175.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251026 21:07:41.451908 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.447562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46747 (local address 127.3.175.2:33931)
1026 21:07:41.448238 (+ 676us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:41.448276 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:41.449012 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:41.449571 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:41.449585 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:41.449680 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:41.450351 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:41.450352 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:41.450394 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:41.450453 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:41.450718 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:41.450726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:41.450733 (+ 7us) client_negotiation.cc:769] Sending connection context
1026 21:07:41.450960 (+ 227us) client_negotiation.cc:240] Negotiation successful
1026 21:07:41.451196 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20251026 21:07:41.453233 14506 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46747
I20251026 21:07:41.453486 14506 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:41.453958 14506 heartbeater.cc:507] Master 127.3.175.62:46747 requested a full tablet report, sending...
I20251026 21:07:41.455004 14187 ts_manager.cc:194] Registered new tserver with Master: 9cc250950a4344c0b37a0dc128dfc0f0 (127.3.175.2:34475)
W20251026 21:07:41.564518 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:41.564834 14509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:41.564917 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:41.568467 14509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:41.568539 14509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:41.568656 14509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:41.572655 14509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:41.573860 14509 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:41.575033 14509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:41.582005 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.582059 14509 server_base.cc:1047] running on GCE node
W20251026 21:07:41.582080 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:41.582230 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:41.582787 14509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:41.583433 14509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:41.584609 14509 hybrid_clock.cc:648] HybridClock initialized: now 1761512861584581 us; error 42 us; skew 500 ppm
I20251026 21:07:41.586571 14509 webserver.cc:492] Webserver started at http://127.3.175.3:43775/ using document root <none> and password file <none>
I20251026 21:07:41.587234 14509 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:41.587330 14509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:41.587559 14509 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:41.589399 14509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "820a725f1226446484635e8aa79a825a"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "83314a4a0068dca7b2d0ef6b475a27d3"
server_key_iv: "99fe08b75f63514e835fcb202421333c"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.589879 14509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "820a725f1226446484635e8aa79a825a"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "83314a4a0068dca7b2d0ef6b475a27d3"
server_key_iv: "99fe08b75f63514e835fcb202421333c"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.593448 14509 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:07:41.595892 14523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.597116 14509 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:41.597259 14509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "820a725f1226446484635e8aa79a825a"
format_stamp: "Formatted at 2025-10-26 21:07:41 on dist-test-slave-th6x"
server_key: "83314a4a0068dca7b2d0ef6b475a27d3"
server_key_iv: "99fe08b75f63514e835fcb202421333c"
server_key_version: "encryptionkey@0"
I20251026 21:07:41.597373 14509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:41.602650 14509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:41.603509 14509 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:41.603713 14509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:41.604317 14509 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:41.605361 14509 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:41.605475 14509 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.605597 14509 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:41.605636 14509 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:41.615048 14636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:43353 every 8 connection(s)
I20251026 21:07:41.615026 14509 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:43353
I20251026 21:07:41.616214 14509 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:41.617938 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14509
I20251026 21:07:41.618057 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:41.618316 3772 external_mini_cluster.cc:1468] Setting key a91b60602a42f68d98fac5416d700df9
I20251026 21:07:41.622004 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.618096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:48545 (local address 127.3.175.62:46747)
1026 21:07:41.618255 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:41.618260 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:41.619436 (+ 1176us) server_negotiation.cc:408] Connection header received
1026 21:07:41.620412 (+ 976us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:41.620415 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:41.620454 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:41.620516 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:41.621617 (+ 1101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:41.621619 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:41.621622 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 21:07:41.621632 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:41.621656 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 21:07:41.621663 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:41.621740 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:41.621848 (+ 108us) server_negotiation.cc:300] Negotiation successful
1026 21:07:41.621892 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 21:07:41.622756 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:41.618493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:46747 (local address 127.3.175.3:48545)
1026 21:07:41.619266 (+ 773us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:41.619342 (+ 76us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:41.620188 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:41.620665 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:41.620674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:41.620766 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:41.621391 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 21:07:41.621393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:41.621425 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 21:07:41.621484 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:41.621734 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:41.621739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:41.621743 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 21:07:41.621930 (+ 187us) client_negotiation.cc:240] Negotiation successful
1026 21:07:41.622126 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":132,"threads_started":1}
I20251026 21:07:41.624071 14637 heartbeater.cc:344] Connected to a master server at 127.3.175.62:46747
I20251026 21:07:41.624334 14637 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:41.624778 14637 heartbeater.cc:507] Master 127.3.175.62:46747 requested a full tablet report, sending...
I20251026 21:07:41.625855 14187 ts_manager.cc:194] Registered new tserver with Master: 820a725f1226446484635e8aa79a825a (127.3.175.3:43353)
I20251026 21:07:41.632737 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 21:07:41.636647 14226 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:07:41.635535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.3.175.62:46747)
1026 21:07:41.635693 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:41.635699 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:41.635858 (+ 159us) server_negotiation.cc:408] Connection header received
1026 21:07:41.636032 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:41.636034 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:41.636088 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:41.636174 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:41.636505 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42324: BlockingRecv error: recv got EOF from 127.0.0.1:42324 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:07:41.636776 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14247
I20251026 21:07:41.644918 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14378
I20251026 21:07:41.651046 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14509
I20251026 21:07:41.657001 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14156
2025-10-26T21:07:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (811 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:41 dist-test-slave-th6x krb5kdc[14647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:41 dist-test-slave-th6x krb5kdc[14647](info): set up 2 sockets
Oct 26 21:07:41 dist-test-slave-th6x krb5kdc[14647](info): commencing operation
krb5kdc: starting...
W20251026 21:07:43.664031 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:43 dist-test-slave-th6x krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:43.802279 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43691
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:43691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:43.913753 14663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:43.914036 14663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:43.914093 14663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:43.917601 14663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:07:43.917675 14663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:43.917699 14663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:43.917724 14663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:07:43.917765 14663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:07:43.922406 14663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:43691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:43691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:43.923614 14663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:43.924542 14663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:43.930625 14671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:43.930624 14669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:43.930734 14663 server_base.cc:1047] running on GCE node
W20251026 21:07:43.930624 14668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:43.931488 14663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:43.932499 14663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:43.933662 14663 hybrid_clock.cc:648] HybridClock initialized: now 1761512863933650 us; error 32 us; skew 500 ppm
Oct 26 21:07:43 dist-test-slave-th6x krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512863, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:43.936625 14663 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:07:43.937754 14663 webserver.cc:492] Webserver started at http://127.3.175.62:42815/ using document root <none> and password file <none>
I20251026 21:07:43.938323 14663 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:43.938371 14663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:43.938535 14663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:43.940326 14663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "0f97bbdcc39947ce87a14f1908ffc0c8"
format_stamp: "Formatted at 2025-10-26 21:07:43 on dist-test-slave-th6x"
server_key: "1feecd5dada8a5d1e8d38dc3c19ce403"
server_key_iv: "511d30b95d4497bfdc3a8c86af984b6a"
server_key_version: "encryptionkey@0"
I20251026 21:07:43.940797 14663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "0f97bbdcc39947ce87a14f1908ffc0c8"
format_stamp: "Formatted at 2025-10-26 21:07:43 on dist-test-slave-th6x"
server_key: "1feecd5dada8a5d1e8d38dc3c19ce403"
server_key_iv: "511d30b95d4497bfdc3a8c86af984b6a"
server_key_version: "encryptionkey@0"
I20251026 21:07:43.944450 14663 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251026 21:07:43.946861 14678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:43.948001 14663 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:43.948110 14663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "0f97bbdcc39947ce87a14f1908ffc0c8"
format_stamp: "Formatted at 2025-10-26 21:07:43 on dist-test-slave-th6x"
server_key: "1feecd5dada8a5d1e8d38dc3c19ce403"
server_key_iv: "511d30b95d4497bfdc3a8c86af984b6a"
server_key_version: "encryptionkey@0"
I20251026 21:07:43.948199 14663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:43.964789 14663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:43.971830 14663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:43.972002 14663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:43.979100 14730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:43691 every 8 connection(s)
I20251026 21:07:43.979113 14663 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:43691
I20251026 21:07:43.980247 14663 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:07:43.983140 14731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:43.988703 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14663
I20251026 21:07:43.988847 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:07:43.989106 3772 external_mini_cluster.cc:1468] Setting key 35c4e77787828ffbc2f9a7e9ebb6ce29
I20251026 21:07:43.989490 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: Bootstrap starting.
I20251026 21:07:43.992064 14731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:43.993012 14731 log.cc:826] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:43.995123 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: No bootstrap required, opened a new log
Oct 26 21:07:43 dist-test-slave-th6x krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:43.998358 14731 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } }
I20251026 21:07:43.998584 14731 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:43.998651 14731 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f97bbdcc39947ce87a14f1908ffc0c8, State: Initialized, Role: FOLLOWER
I20251026 21:07:43.999146 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } }
I20251026 21:07:43.999320 14731 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:07:43.999397 14731 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:07:43.999496 14731 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:44.000666 14731 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } }
I20251026 21:07:44.001034 14731 leader_election.cc:304] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f97bbdcc39947ce87a14f1908ffc0c8; no voters:
I20251026 21:07:44.001366 14731 leader_election.cc:290] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:07:44.001911 14736 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:44.002120 14736 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0f97bbdcc39947ce87a14f1908ffc0c8, State: Running, Role: LEADER
I20251026 21:07:44.002427 14736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } }
I20251026 21:07:44.002622 14731 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:07:44.002880 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:43.990483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.3.175.62:43691)
1026 21:07:43.990997 (+ 514us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:43.991008 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:43.991045 (+ 37us) server_negotiation.cc:408] Connection header received
1026 21:07:43.991632 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:43.991652 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:43.991974 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:43.992311 (+ 337us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:43.993383 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:43.994148 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:43.994997 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:43.995220 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:43.997645 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:43.997666 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:43.997677 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:43.997712 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:43.999769 (+ 2057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.000296 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.000302 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.000306 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.000388 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.000663 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.000665 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.000667 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.001066 (+ 399us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:44.001297 (+ 231us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.001670 (+ 373us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.001964 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":173,"threads_started":1}
I20251026 21:07:44.004032 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } } }
I20251026 21:07:44.004163 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:44.004474 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f97bbdcc39947ce87a14f1908ffc0c8. Latest consensus state: current_term: 1 leader_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f97bbdcc39947ce87a14f1908ffc0c8" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 43691 } } }
I20251026 21:07:44.004562 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8 [sys.catalog]: This master's current role is: LEADER
I20251026 21:07:44.005462 14745 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:07:44.008122 14745 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:07:44.013784 14745 catalog_manager.cc:1357] Generated new cluster ID: 7e40e95b91ee417496bd49050d62236d
I20251026 21:07:44.013866 14745 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:07:44.040112 14745 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:07:44.041127 14745 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:07:44.051677 14745 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: Generated new TSK 0
I20251026 21:07:44.052489 14745 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:44.104586 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:44.219244 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:44.219550 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:44.219616 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:44.223091 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:44.223204 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:44.223331 14759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:07:44.227927 14759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:44.229096 14759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:44.230017 14759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:44.236843 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.236846 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.236845 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:44.237176 14759 server_base.cc:1047] running on GCE node
I20251026 21:07:44.237586 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:44.238158 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:44.239374 14759 hybrid_clock.cc:648] HybridClock initialized: now 1761512864239347 us; error 40 us; skew 500 ppm
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:44.242432 14759 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:07:44.243634 14759 webserver.cc:492] Webserver started at http://127.3.175.1:39979/ using document root <none> and password file <none>
I20251026 21:07:44.244246 14759 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:44.244307 14759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:44.244503 14759 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:44.246294 14759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "46ad19771b054ef2971323eb7c2cfed6"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "4aa7d5b2e8508e0b47f3265b97668a2f"
server_key_iv: "3caef8c298aaf3f86ce5cc32a6ee20ed"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.246806 14759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "46ad19771b054ef2971323eb7c2cfed6"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "4aa7d5b2e8508e0b47f3265b97668a2f"
server_key_iv: "3caef8c298aaf3f86ce5cc32a6ee20ed"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.250532 14759 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251026 21:07:44.253067 14774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.254123 14759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 21:07:44.254241 14759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "46ad19771b054ef2971323eb7c2cfed6"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "4aa7d5b2e8508e0b47f3265b97668a2f"
server_key_iv: "3caef8c298aaf3f86ce5cc32a6ee20ed"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.254345 14759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:44.268240 14759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:44.271745 14759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:44.271956 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:44.272558 14759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:44.273514 14759 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:44.273569 14759 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.273631 14759 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:44.273665 14759 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.282824 14759 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:38293
I20251026 21:07:44.282850 14887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:38293 every 8 connection(s)
I20251026 21:07:44.283897 14759 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:07:44.291074 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14759
I20251026 21:07:44.291229 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:07:44.291522 3772 external_mini_cluster.cc:1468] Setting key 608dff98c27aa4216dd90c71bd4ca005
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:44.297685 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.285776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:58841 (local address 127.3.175.62:43691)
1026 21:07:44.285938 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.285942 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.286646 (+ 704us) server_negotiation.cc:408] Connection header received
1026 21:07:44.287599 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.287603 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.287674 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.287753 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:44.289148 (+ 1395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.289683 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.290372 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.290573 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.293755 (+ 3182us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:44.293776 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:44.293778 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:44.293804 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:44.295595 (+ 1791us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.296129 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.296133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.296135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.296197 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.296491 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.296494 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.296497 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.296672 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:44.296767 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.297353 (+ 586us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.297506 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 21:07:44.298406 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.286037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43691 (local address 127.3.175.1:58841)
1026 21:07:44.286492 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.286530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.287382 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.287957 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.287966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.288389 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:44.288970 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.288983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.289816 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.289820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.290236 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.290244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.290498 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.291795 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:44.291823 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:44.293581 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:44.295726 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.295733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.295748 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.296021 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.296295 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.296297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.296299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.296398 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.296783 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:44.296792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:44.297070 (+ 278us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.297304 (+ 234us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.297611 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20251026 21:07:44.299872 14888 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43691
I20251026 21:07:44.300141 14888 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:07:44.300722 14888 heartbeater.cc:507] Master 127.3.175.62:43691 requested a full tablet report, sending...
I20251026 21:07:44.302310 14695 ts_manager.cc:194] Registered new tserver with Master: 46ad19771b054ef2971323eb7c2cfed6 (127.3.175.1:38293)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:07:44.303701 14695 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:58841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:44.340435 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:44.453420 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:44.453683 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:44.453745 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:44.457155 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:44.457231 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:44.457321 14895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:07:44.462020 14895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:44.463236 14895 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:44.464149 14895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:44.471099 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.471102 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.471099 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:44.471520 14895 server_base.cc:1047] running on GCE node
I20251026 21:07:44.471915 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:44.472488 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:44.473644 14895 hybrid_clock.cc:648] HybridClock initialized: now 1761512864473632 us; error 29 us; skew 500 ppm
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:44.476735 14895 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:07:44.477993 14895 webserver.cc:492] Webserver started at http://127.3.175.2:37007/ using document root <none> and password file <none>
I20251026 21:07:44.478633 14895 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:44.478689 14895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:44.478896 14895 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:44.480803 14895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "8048a26daf854eb58284400f9dcec62b"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "5bf8ce9d4d3e1b279e7cbc2c441a6ac0"
server_key_iv: "a28dc93f93b3c4b1b43804c1465c93ec"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.481315 14895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "8048a26daf854eb58284400f9dcec62b"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "5bf8ce9d4d3e1b279e7cbc2c441a6ac0"
server_key_iv: "a28dc93f93b3c4b1b43804c1465c93ec"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.484997 14895 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251026 21:07:44.487386 14910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.488457 14895 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 21:07:44.488579 14895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "8048a26daf854eb58284400f9dcec62b"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "5bf8ce9d4d3e1b279e7cbc2c441a6ac0"
server_key_iv: "a28dc93f93b3c4b1b43804c1465c93ec"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.488677 14895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:44.526667 14895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:44.529747 14895 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:44.529976 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:44.530575 14895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:44.531519 14895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:44.531572 14895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.531634 14895 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:44.531667 14895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.540858 14895 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:38589
I20251026 21:07:44.540889 15023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:38589 every 8 connection(s)
I20251026 21:07:44.541925 14895 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:07:44.547359 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 14895
I20251026 21:07:44.547497 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:07:44.547749 3772 external_mini_cluster.cc:1468] Setting key 71d2e4b76714310db45696066e3040ea
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:44.555370 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.543806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.2:57577 (local address 127.3.175.62:43691)
1026 21:07:44.543961 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.543965 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.544711 (+ 746us) server_negotiation.cc:408] Connection header received
1026 21:07:44.545561 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.545565 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.545619 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.545697 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:44.547024 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.547652 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.548455 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.548634 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.551615 (+ 2981us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:44.551634 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:44.551636 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:44.551666 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:44.553317 (+ 1651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.553849 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.553853 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.553854 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.553903 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.554195 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.554198 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.554202 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.554363 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:44.554456 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.555010 (+ 554us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.555150 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251026 21:07:44.556077 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.544077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43691 (local address 127.3.175.2:57577)
1026 21:07:44.544550 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.544588 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.545365 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.545846 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.545854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.546300 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:44.546880 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.546893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.547787 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.547792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.548329 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.548341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.548589 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.549766 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:44.549795 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:44.551433 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:44.553452 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.553459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.553473 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.553740 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.553996 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.553999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.554000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.554099 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.554472 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:44.554481 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:44.554740 (+ 259us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.554976 (+ 236us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.555305 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.3.175.3@KRBTEST.COM; defaulting to no policy
I20251026 21:07:44.557224 15024 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43691
I20251026 21:07:44.557503 15024 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:44.558089 15024 heartbeater.cc:507] Master 127.3.175.62:43691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.3@KRBTEST.COM" created.
I20251026 21:07:44.559368 14695 ts_manager.cc:194] Registered new tserver with Master: 8048a26daf854eb58284400f9dcec62b (127.3.175.2:38589)
I20251026 21:07:44.560189 14695 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.2@KRBTEST.COM'} at 127.3.175.2:57577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:07:44.597875 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.3:0
--local_ip_for_outbound_sockets=127.3.175.3
--webserver_interface=127.3.175.3
--webserver_port=0
--tserver_master_addrs=127.3.175.62:43691
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:07:44.709940 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:07:44.710227 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:07:44.710325 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:07:44.713799 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:07:44.713896 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:07:44.714026 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.3
I20251026 21:07:44.718492 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.175.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:43691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.3
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:07:44.719719 15031 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:07:44.720611 15031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:07:44.727571 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.727636 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:07:44.727574 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:07:44.727860 15031 server_base.cc:1047] running on GCE node
I20251026 21:07:44.728298 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:07:44.728894 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:07:44.730055 15031 hybrid_clock.cc:648] HybridClock initialized: now 1761512864730038 us; error 42 us; skew 500 ppm
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:07:44.733053 15031 init.cc:377] Logged in from keytab as kudu/127.3.175.3@KRBTEST.COM (short username kudu)
I20251026 21:07:44.734198 15031 webserver.cc:492] Webserver started at http://127.3.175.3:45845/ using document root <none> and password file <none>
I20251026 21:07:44.734781 15031 fs_manager.cc:362] Metadata directory not provided
I20251026 21:07:44.734840 15031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:07:44.735030 15031 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 21:07:44.736815 15031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/instance:
uuid: "af93b1c996ff49cc94b06069daada6e3"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "aa12cfbfec18ba70949f1ee4bae406be"
server_key_iv: "dc09954380ba0d2efa3878103eb5b880"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.737319 15031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance:
uuid: "af93b1c996ff49cc94b06069daada6e3"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "aa12cfbfec18ba70949f1ee4bae406be"
server_key_iv: "dc09954380ba0d2efa3878103eb5b880"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.740934 15031 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251026 21:07:44.743526 15046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.744618 15031 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 21:07:44.744737 15031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
uuid: "af93b1c996ff49cc94b06069daada6e3"
format_stamp: "Formatted at 2025-10-26 21:07:44 on dist-test-slave-th6x"
server_key: "aa12cfbfec18ba70949f1ee4bae406be"
server_key_iv: "dc09954380ba0d2efa3878103eb5b880"
server_key_version: "encryptionkey@0"
I20251026 21:07:44.744846 15031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:07:44.757198 15031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:07:44.760330 15031 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:07:44.760526 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:07:44.761127 15031 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:07:44.762081 15031 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:07:44.762130 15031 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.762189 15031 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:07:44.762221 15031 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:07:44.771466 15031 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.3:33485
I20251026 21:07:44.771498 15159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.3:33485 every 8 connection(s)
I20251026 21:07:44.772482 15031 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/data/info.pb
I20251026 21:07:44.773958 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 15031
I20251026 21:07:44.774101 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/instance
I20251026 21:07:44.774410 3772 external_mini_cluster.cc:1468] Setting key 8038e595c632905abeb534ce90ce2c94
Oct 26 21:07:44 dist-test-slave-th6x krb5kdc[14647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.3@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:07:44.785491 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.774377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.3:46961 (local address 127.3.175.62:43691)
1026 21:07:44.774517 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.774520 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.775418 (+ 898us) server_negotiation.cc:408] Connection header received
1026 21:07:44.776382 (+ 964us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.776386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.776449 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.776528 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:44.777868 (+ 1340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.778388 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.779036 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.779236 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.782051 (+ 2815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:44.782073 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:44.782076 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:44.782106 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:44.783499 (+ 1393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.784036 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.784039 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.784041 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.784086 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.784329 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.784331 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.784332 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.784492 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:44.784565 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.785141 (+ 576us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.785313 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 21:07:44.786231 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.774767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:43691 (local address 127.3.175.3:46961)
1026 21:07:44.775248 (+ 481us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.775321 (+ 73us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.776161 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.776710 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.776719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.777136 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:07:44.777704 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.777716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.778514 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.778517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.778911 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.778917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.779661 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.780285 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:07:44.780310 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:07:44.781850 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:07:44.783640 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.783646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.783657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.783948 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.784166 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:07:44.784168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:07:44.784170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:07:44.784264 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:07:44.784570 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:07:44.784575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:07:44.784823 (+ 248us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.785072 (+ 249us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.785380 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20251026 21:07:44.787536 15160 heartbeater.cc:344] Connected to a master server at 127.3.175.62:43691
I20251026 21:07:44.787824 15160 heartbeater.cc:461] Registering TS with master...
I20251026 21:07:44.788482 15160 heartbeater.cc:507] Master 127.3.175.62:43691 requested a full tablet report, sending...
I20251026 21:07:44.789469 14695 ts_manager.cc:194] Registered new tserver with Master: af93b1c996ff49cc94b06069daada6e3 (127.3.175.3:33485)
I20251026 21:07:44.790059 14695 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.3@KRBTEST.COM'} at 127.3.175.3:46961
I20251026 21:07:44.800509 3772 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 21:07:44.809156 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.802375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.3.175.62:43691)
1026 21:07:44.802513 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.802517 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.802682 (+ 165us) server_negotiation.cc:408] Connection header received
1026 21:07:44.802823 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.802826 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.802875 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.802949 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:07:44.803926 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.804447 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.805137 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.805370 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.806420 (+ 1050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:07:44.806441 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:07:44.806444 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:07:44.806470 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:07:44.807693 (+ 1223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.808085 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.808088 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.808090 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.808135 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:07:44.808351 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:07:44.808354 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:07:44.808356 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:07:44.808534 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:07:44.808658 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.808869 (+ 211us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.808995 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 21:07:44.811816 14695 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 21:07:44.814284 14695 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 21:07:44.828212 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.823865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:33485 (local address 127.0.0.1:46716)
1026 21:07:44.824834 (+ 969us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.824849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.824938 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.825375 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.825378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.825399 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:44.825644 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.825650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.826815 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.826819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.827646 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.827655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.827761 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.827842 (+ 81us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.827944 (+ 102us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.828043 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":110,"threads_started":1}
I20251026 21:07:44.828306 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.822961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.3.175.2:38589)
1026 21:07:44.823611 (+ 650us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.823621 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.823650 (+ 29us) server_negotiation.cc:408] Connection header received
1026 21:07:44.823863 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.823871 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.824081 (+ 210us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.824227 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:44.824984 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.825962 (+ 978us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.827040 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.827624 (+ 584us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.827761 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.828038 (+ 277us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.828144 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":156,"threads_started":1}
I20251026 21:07:44.828717 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.822813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:38589 (local address 127.0.0.1:49168)
1026 21:07:44.823445 (+ 632us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.823487 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.823631 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.824455 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.824461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.824490 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:44.824859 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.824867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.826065 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.826068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.826870 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.826882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.827824 (+ 942us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.827845 (+ 21us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.828520 (+ 675us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.828590 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":114,"threads_started":1}
I20251026 21:07:44.828717 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.823494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.3.175.1:38293)
1026 21:07:44.823824 (+ 330us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.823831 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.824064 (+ 233us) server_negotiation.cc:408] Connection header received
1026 21:07:44.824208 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.824212 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.824333 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.824454 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:44.825208 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.826228 (+ 1020us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.827605 (+ 1377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.828165 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.828268 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.828513 (+ 245us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.828605 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20251026 21:07:44.828734 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.824285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.3.175.3:33485)
1026 21:07:44.824849 (+ 564us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.824856 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.824912 (+ 56us) server_negotiation.cc:408] Connection header received
1026 21:07:44.825086 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.825089 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.825237 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.825387 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:44.825769 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.826700 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.827782 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.828322 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.828443 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.828506 (+ 63us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.828584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":138,"threads_started":1}
I20251026 21:07:44.829124 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.823048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:38293 (local address 127.0.0.1:53760)
1026 21:07:44.823953 (+ 905us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.823969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.824073 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.824457 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.824461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.824513 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:44.825078 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.825086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.826354 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.826357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.827447 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.827458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.828339 (+ 881us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.828357 (+ 18us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.828943 (+ 586us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.829007 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"spinlock_wait_cycles":32512,"thread_start_us":111,"threads_started":1}
I20251026 21:07:44.831710 14822 tablet_service.cc:1505] Processing CreateTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 (DEFAULT_TABLE table=test-table [id=5640275dfb8c4b48a549f9c41c6342db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:44.831717 15094 tablet_service.cc:1505] Processing CreateTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 (DEFAULT_TABLE table=test-table [id=5640275dfb8c4b48a549f9c41c6342db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:44.831717 14958 tablet_service.cc:1505] Processing CreateTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 (DEFAULT_TABLE table=test-table [id=5640275dfb8c4b48a549f9c41c6342db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 21:07:44.832633 14822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb71c1c7966488f95164b82aeb9abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:44.832630 15094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb71c1c7966488f95164b82aeb9abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:44.832630 14958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb71c1c7966488f95164b82aeb9abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:07:44.838526 15177 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: Bootstrap starting.
I20251026 21:07:44.839805 15179 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Bootstrap starting.
I20251026 21:07:44.839984 15178 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Bootstrap starting.
I20251026 21:07:44.841527 15177 tablet_bootstrap.cc:654] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:44.842149 15179 tablet_bootstrap.cc:654] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:44.842149 15178 tablet_bootstrap.cc:654] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Neither blocks nor log segments found. Creating new log.
I20251026 21:07:44.842311 15177 log.cc:826] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:44.843012 15178 log.cc:826] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:44.843012 15179 log.cc:826] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Log is configured to *not* fsync() on all Append() calls
I20251026 21:07:44.844297 15177 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: No bootstrap required, opened a new log
I20251026 21:07:44.844498 15177 ts_tablet_manager.cc:1403] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 21:07:44.844798 15178 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: No bootstrap required, opened a new log
I20251026 21:07:44.845088 15178 ts_tablet_manager.cc:1403] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:07:44.845444 15179 tablet_bootstrap.cc:492] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: No bootstrap required, opened a new log
I20251026 21:07:44.845630 15179 ts_tablet_manager.cc:1403] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 21:07:44.847571 15177 raft_consensus.cc:359] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.847781 15177 raft_consensus.cc:385] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:44.847862 15177 raft_consensus.cc:740] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ad19771b054ef2971323eb7c2cfed6, State: Initialized, Role: FOLLOWER
I20251026 21:07:44.848357 15177 consensus_queue.cc:260] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.849182 15177 ts_tablet_manager.cc:1434] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:07:44.849066 15178 raft_consensus.cc:359] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.849277 15178 raft_consensus.cc:385] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:44.849386 15178 raft_consensus.cc:740] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b1c996ff49cc94b06069daada6e3, State: Initialized, Role: FOLLOWER
I20251026 21:07:44.849598 14888 heartbeater.cc:499] Master 127.3.175.62:43691 was elected leader, sending a full tablet report...
I20251026 21:07:44.849589 15179 raft_consensus.cc:359] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.849835 15179 raft_consensus.cc:385] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:07:44.849895 15179 raft_consensus.cc:740] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048a26daf854eb58284400f9dcec62b, State: Initialized, Role: FOLLOWER
I20251026 21:07:44.849979 15178 consensus_queue.cc:260] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.850308 15179 consensus_queue.cc:260] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.850769 15160 heartbeater.cc:499] Master 127.3.175.62:43691 was elected leader, sending a full tablet report...
I20251026 21:07:44.851068 15178 ts_tablet_manager.cc:1434] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 21:07:44.851151 15179 ts_tablet_manager.cc:1434] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 21:07:44.851516 15024 heartbeater.cc:499] Master 127.3.175.62:43691 was elected leader, sending a full tablet report...
I20251026 21:07:44.966666 15183 raft_consensus.cc:493] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 21:07:44.966885 15183 raft_consensus.cc:515] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.968232 15183 leader_election.cc:290] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8048a26daf854eb58284400f9dcec62b (127.3.175.2:38589), af93b1c996ff49cc94b06069daada6e3 (127.3.175.3:33485)
I20251026 21:07:44.972047 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.968962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.3:33485 (local address 127.3.175.1:49389)
1026 21:07:44.969264 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.969277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.969380 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.969642 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.969644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.969660 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:44.969953 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.969961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.970846 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.970855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.971639 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.971649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.971756 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.971777 (+ 21us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.971815 (+ 38us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.971878 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
I20251026 21:07:44.972263 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.968489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.2:38589 (local address 127.3.175.1:49385)
1026 21:07:44.968876 (+ 387us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:44.968889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 21:07:44.969059 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:07:44.969494 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:07:44.969497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:07:44.969518 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 21:07:44.970237 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.970249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.971141 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.971144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:07:44.971771 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:07:44.971780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.971899 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.971918 (+ 19us) client_negotiation.cc:769] Sending connection context
1026 21:07:44.971981 (+ 63us) client_negotiation.cc:240] Negotiation successful
1026 21:07:44.972036 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":161,"threads_started":1}
I20251026 21:07:44.972596 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.969106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:49389 (local address 127.3.175.3:33485)
1026 21:07:44.969261 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.969267 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.969337 (+ 70us) server_negotiation.cc:408] Connection header received
1026 21:07:44.969487 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.969490 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.969540 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.969625 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:44.970086 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.970709 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.971770 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.972296 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.972344 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.972405 (+ 61us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.972482 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251026 21:07:44.972702 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:44.968646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:49385 (local address 127.3.175.2:38589)
1026 21:07:44.968818 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:44.968823 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:44.969020 (+ 197us) server_negotiation.cc:408] Connection header received
1026 21:07:44.969216 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:44.969220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:44.969279 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:44.969337 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 21:07:44.970407 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.971013 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:44.971912 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:44.972405 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:44.972447 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:44.972503 (+ 56us) server_negotiation.cc:300] Negotiation successful
1026 21:07:44.972566 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 21:07:44.973268 14978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fb71c1c7966488f95164b82aeb9abe9" candidate_uuid: "46ad19771b054ef2971323eb7c2cfed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048a26daf854eb58284400f9dcec62b" is_pre_election: true
I20251026 21:07:44.973271 15114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fb71c1c7966488f95164b82aeb9abe9" candidate_uuid: "46ad19771b054ef2971323eb7c2cfed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b1c996ff49cc94b06069daada6e3" is_pre_election: true
I20251026 21:07:44.973587 14978 raft_consensus.cc:2468] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46ad19771b054ef2971323eb7c2cfed6 in term 0.
I20251026 21:07:44.973587 15114 raft_consensus.cc:2468] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46ad19771b054ef2971323eb7c2cfed6 in term 0.
I20251026 21:07:44.974102 14778 leader_election.cc:304] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46ad19771b054ef2971323eb7c2cfed6, af93b1c996ff49cc94b06069daada6e3; no voters:
I20251026 21:07:44.974418 15183 raft_consensus.cc:2804] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 21:07:44.974500 15183 raft_consensus.cc:493] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 21:07:44.974550 15183 raft_consensus.cc:3060] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:44.975533 15183 raft_consensus.cc:515] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.975873 15183 leader_election.cc:290] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [CANDIDATE]: Term 1 election: Requested vote from peers 8048a26daf854eb58284400f9dcec62b (127.3.175.2:38589), af93b1c996ff49cc94b06069daada6e3 (127.3.175.3:33485)
I20251026 21:07:44.976274 14978 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fb71c1c7966488f95164b82aeb9abe9" candidate_uuid: "46ad19771b054ef2971323eb7c2cfed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048a26daf854eb58284400f9dcec62b"
I20251026 21:07:44.976323 15114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fb71c1c7966488f95164b82aeb9abe9" candidate_uuid: "46ad19771b054ef2971323eb7c2cfed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b1c996ff49cc94b06069daada6e3"
I20251026 21:07:44.976408 14978 raft_consensus.cc:3060] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:44.976436 15114 raft_consensus.cc:3060] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:07:44.977380 14978 raft_consensus.cc:2468] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46ad19771b054ef2971323eb7c2cfed6 in term 1.
I20251026 21:07:44.977385 15114 raft_consensus.cc:2468] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46ad19771b054ef2971323eb7c2cfed6 in term 1.
I20251026 21:07:44.977674 14778 leader_election.cc:304] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46ad19771b054ef2971323eb7c2cfed6, 8048a26daf854eb58284400f9dcec62b; no voters:
I20251026 21:07:44.977845 15183 raft_consensus.cc:2804] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:07:44.978092 15183 raft_consensus.cc:697] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [term 1 LEADER]: Becoming Leader. State: Replica: 46ad19771b054ef2971323eb7c2cfed6, State: Running, Role: LEADER
I20251026 21:07:44.978438 15183 consensus_queue.cc:237] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } }
I20251026 21:07:44.981794 14693 catalog_manager.cc:5649] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46ad19771b054ef2971323eb7c2cfed6 (127.3.175.1). New cstate: current_term: 1 leader_uuid: "46ad19771b054ef2971323eb7c2cfed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46ad19771b054ef2971323eb7c2cfed6" member_type: VOTER last_known_addr { host: "127.3.175.1" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 } health_report { overall_health: UNKNOWN } } }
I20251026 21:07:45.020869 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:45.017920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.3.175.1:38293)
1026 21:07:45.018056 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 21:07:45.018060 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 21:07:45.018074 (+ 14us) server_negotiation.cc:408] Connection header received
1026 21:07:45.018156 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:07:45.018159 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:07:45.018218 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:07:45.018303 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 21:07:45.018619 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:45.019122 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:07:45.019765 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:07:45.019910 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:07:45.020019 (+ 109us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 21:07:45.020480 (+ 461us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 21:07:45.020578 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 21:07:45.020645 (+ 67us) server_negotiation.cc:300] Negotiation successful
1026 21:07:45.020696 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20251026 21:07:45.023370 15161 tablet.cc:2378] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 21:07:45.026765 15114 raft_consensus.cc:1275] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 1 FOLLOWER]: Refusing update from remote peer 46ad19771b054ef2971323eb7c2cfed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:45.026765 14978 raft_consensus.cc:1275] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 1 FOLLOWER]: Refusing update from remote peer 46ad19771b054ef2971323eb7c2cfed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 21:07:45.027544 15183 consensus_queue.cc:1048] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af93b1c996ff49cc94b06069daada6e3" member_type: VOTER last_known_addr { host: "127.3.175.3" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:45.027787 15188 consensus_queue.cc:1048] T 4fb71c1c7966488f95164b82aeb9abe9 P 46ad19771b054ef2971323eb7c2cfed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8048a26daf854eb58284400f9dcec62b" member_type: VOTER last_known_addr { host: "127.3.175.2" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 21:07:45.035992 15192 mvcc.cc:204] Tried to move back new op lower bound from 7215156695141707776 to 7215156694954708992. Current Snapshot: MvccSnapshot[applied={T|T < 7215156695141707776}]
I20251026 21:07:45.038527 15193 mvcc.cc:204] Tried to move back new op lower bound from 7215156695141707776 to 7215156694954708992. Current Snapshot: MvccSnapshot[applied={T|T < 7215156695141707776 or (T in {7215156695141707776})}]
I20251026 21:07:45.043756 14694 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:45.043933 14694 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 21:07:45.046337 14694 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0f97bbdcc39947ce87a14f1908ffc0c8: Sending DeleteTablet for 3 replicas of tablet 4fb71c1c7966488f95164b82aeb9abe9
I20251026 21:07:45.047163 14822 tablet_service.cc:1552] Processing DeleteTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:45 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:53760
I20251026 21:07:45.047382 14958 tablet_service.cc:1552] Processing DeleteTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:45 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:49168
I20251026 21:07:45.047399 15094 tablet_service.cc:1552] Processing DeleteTablet for tablet 4fb71c1c7966488f95164b82aeb9abe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 21:07:45 UTC) from {username='kudu', principal='kudu/127.3.175.62@KRBTEST.COM'} at 127.0.0.1:46716
I20251026 21:07:45.047811 15202 tablet_replica.cc:333] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: stopping tablet replica
I20251026 21:07:45.048084 15202 raft_consensus.cc:2243] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:45.048398 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14759
I20251026 21:07:45.048451 15202 raft_consensus.cc:2272] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:45.050606 15203 tablet_replica.cc:333] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: stopping tablet replica
I20251026 21:07:45.050873 15203 raft_consensus.cc:2243] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 21:07:45.051175 15203 raft_consensus.cc:2272] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 21:07:45.051406 15202 ts_tablet_manager.cc:1916] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:45.054819 15202 ts_tablet_manager.cc:1929] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:45.054940 15202 log.cc:1199] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-1/wal/wals/4fb71c1c7966488f95164b82aeb9abe9
I20251026 21:07:45.055423 15202 ts_tablet_manager.cc:1950] T 4fb71c1c7966488f95164b82aeb9abe9 P 8048a26daf854eb58284400f9dcec62b: Deleting consensus metadata
I20251026 21:07:45.056109 15203 ts_tablet_manager.cc:1916] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 21:07:45.056895 14682 catalog_manager.cc:4985] TS 8048a26daf854eb58284400f9dcec62b (127.3.175.2:38589): tablet 4fb71c1c7966488f95164b82aeb9abe9 (table test-table [id=5640275dfb8c4b48a549f9c41c6342db]) successfully deleted
W20251026 21:07:45.057142 14680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.175.1:38293 (error 108)
I20251026 21:07:45.057945 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14895
I20251026 21:07:45.058274 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:07:45.057950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.1:38293 (local address 127.0.0.1:53774)
1026 21:07:45.058096 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 21:07:45.058162 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.1:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251026 21:07:45.058943 14680 catalog_manager.cc:4712] TS 46ad19771b054ef2971323eb7c2cfed6 (127.3.175.1:38293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fb71c1c7966488f95164b82aeb9abe9: Network error: Client connection negotiation failed: client connection to 127.3.175.1:38293: connect: Connection refused (error 111)
I20251026 21:07:45.059118 15203 ts_tablet_manager.cc:1929] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 21:07:45.059223 15203 log.cc:1199] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Deleting WAL directory at /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761512762855912-3772-0/minicluster-data/ts-2/wal/wals/4fb71c1c7966488f95164b82aeb9abe9
I20251026 21:07:45.059671 15203 ts_tablet_manager.cc:1950] T 4fb71c1c7966488f95164b82aeb9abe9 P af93b1c996ff49cc94b06069daada6e3: Deleting consensus metadata
I20251026 21:07:45.061143 14682 catalog_manager.cc:4985] TS af93b1c996ff49cc94b06069daada6e3 (127.3.175.3:33485): tablet 4fb71c1c7966488f95164b82aeb9abe9 (table test-table [id=5640275dfb8c4b48a549f9c41c6342db]) successfully deleted
I20251026 21:07:45.066191 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 15031
I20251026 21:07:45.072327 3772 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task7MLemc/build/debug/bin/kudu with pid 14663
2025-10-26T21:07:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3416 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 21:07:45 dist-test-slave-th6x krb5kdc[15207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 21:07:45 dist-test-slave-th6x krb5kdc[15207](info): set up 2 sockets
Oct 26 21:07:45 dist-test-slave-th6x krb5kdc[15207](info): commencing operation
krb5kdc: starting...
W20251026 21:07:47.084064 3772 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/test-user.keytab.
Oct 26 21:07:47 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T21:07:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T21:07:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.175.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.175.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangeradmin_127.3.175.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangeradmin_127.3.175.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.175.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.175.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangerlookup_127.3.175.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangerlookup_127.3.175.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.175.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/HTTP_127.3.175.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/HTTP_127.3.175.20@KRBTEST.COM.keytab.
I20251026 21:07:47.250325 3772 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7MLemc/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/postgres -l logfile start
2025-10-26 21:07:49.972 UTC [15238] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-26 21:07:49.972 UTC [15238] LOG: listening on IPv4 address "127.3.175.20", port 55143
2025-10-26 21:07:49.976 UTC [15238] LOG: listening on Unix socket "/tmp/.s.PGSQL.55143"
2025-10-26 21:07:49.984 UTC [15243] LOG: database system was shut down at 2025-10-26 21:07:48 UTC
2025-10-26 21:07:49.991 UTC [15238] LOG: database system is ready to accept connections
I20251026 21:07:51.900154 3772 mini_postgres.cc:96] Postgres bound to 55143
2025-10-26 21:07:51.906 UTC [15249] FATAL: database "slave" does not exist
127.3.175.20:55143 - accepting connections
I20251026 21:07:51.907145 3772 mini_ranger.cc:161] Starting Ranger...
I20251026 21:07:51.925927 3772 mini_ranger.cc:84] Created miniranger Postgres user
I20251026 21:07:51.996153 3772 mini_ranger.cc:87] Created ranger Postgres database
I20251026 21:07:51.996291 3772 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin
2025-10-26 21:07:52,291 [I] DB FLAVOR :POSTGRES
2025-10-26 21:07:52,291 [I] --------- Verifying Ranger DB connection ---------
2025-10-26 21:07:52,291 [I] Checking connection..
2025-10-26 21:07:52,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-26 21:07:52,624 [I] Checking connection passed.
2025-10-26 21:07:52,624 [I] --------- Verifying version history table ---------
2025-10-26 21:07:52,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 21:07:52,967 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 21:07:52,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 21:07:53,280 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 21:07:53,281 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-26 21:07:53,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-26 21:07:53.608 UTC [15316] WARNING: there is no transaction in progress
2025-10-26 21:07:53,615 [I] create_dbversion_catalog.sql file imported successfully
2025-10-26 21:07:53,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 21:07:53,960 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-26 21:07:53,961 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-26 21:07:53,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-26 21:07:54,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-26 21:07:54,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-th6x.c.gcp-upstream.internal','N') ;"
2025-10-26 21:07:54,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 21:07:55,282 [I] Table x_portal_user does not exist in database ranger
2025-10-26 21:07:55,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 21:07:55,613 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-26 21:07:55,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 21:07:55,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 21:07:56,282 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-26 21:07:56,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-26 21:07:57.180 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.195 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.211 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.226 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.239 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.252 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.302 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.308 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.318 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.329 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.339 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.351 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.359 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.365 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.369 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57.373 UTC [15433] WARNING: there is no transaction in progress
2025-10-26 21:07:57,811 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-26 21:07:57,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-26 21:07:58,133 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-26 21:07:58,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 21:07:58,489 [I] Table x_portal_user already exists in database 'ranger'
2025-10-26 21:07:58,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 21:07:58,835 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-26 21:07:58,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 21:07:59,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 21:07:59,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-th6x.c.gcp-upstream.internal';"
2025-10-26 21:07:59,789 [I] CORE_DB_SCHEMA import status has been updated
2025-10-26 21:07:59,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-26 21:08:00,129 [I] DB_PATCHES have already been applied
I20251026 21:08:00.134661 3772 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7MLemc/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251026 21:08:00.134727 3772 mini_ranger.cc:193] Using host: 127.3.175.20
I20251026 21:08:00.137825 3772 mini_ranger.cc:227] Ranger admin URL: http://127.3.175.20:51599
Oct 26, 2025 9:08:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 26, 2025 9:08:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 26, 2025 9:08:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin/ews/webapp
Oct 26, 2025 9:08:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 26, 2025 9:08:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 26, 2025 9:08:00 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 26, 2025 9:08:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-admin/ews/webapp].
Oct 26, 2025 9:08:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 26, 2025 9:08:01 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-51599"]
Oct 26, 2025 9:08:01 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 26, 2025 9:08:01 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 26, 2025 9:08:01 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 26, 2025 9:08:01 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 26, 2025 9:08:01 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251026 21:08:02.360150 3772 mini_ranger.cc:160] Time spent starting Ranger: real 10.453s user 0.000s sys 0.011s
Oct 26, 2025 9:08:02 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 26, 2025 9:08:02 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 26, 2025 9:08:02 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 26, 2025 9:08:02 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 26, 2025 9:08:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 26, 2025 9:08:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 26, 2025 9:08:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 26, 2025 9:08:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 26, 2025 9:08:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 26, 2025 9:08:24 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-51599"]
I20251026 21:08:24.537165 3772 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.175.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.175.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangerkms_127.3.175.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.175.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/rangerkms_127.3.175.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.175.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.175.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/HTTP_127.3.175.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.175.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/HTTP_127.3.175.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 26 21:08:24 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251026 21:08:24.622690 3772 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251026 21:08:24.634434 3772 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251026 21:08:24.742604 3772 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251026 21:08:24.742723 3772 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms
I20251026 21:08:24.742764 3772 mini_ranger_kms.cc:227] Using postgres at 127.3.175.20:55143
2025-10-26 21:08:24,954 [I] DB FLAVOR :POSTGRES
2025-10-26 21:08:24,955 [I] --------- Verifying Ranger DB connection ---------
2025-10-26 21:08:24,955 [I] Checking connection
2025-10-26 21:08:24,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-26 21:08:25,300 [I] connection success
2025-10-26 21:08:25,301 [I] --------- Verifying Ranger DB tables ---------
2025-10-26 21:08:25,301 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-26 21:08:25,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-26 21:08:25,635 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-26 21:08:25,636 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-26 21:08:25,636 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-26 21:08:25,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.175.20:55143/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-26 21:08:25,963 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251026 21:08:26.498483 3772 mini_ranger_kms.cc:310] Created kms service
I20251026 21:08:26.722347 3772 mini_ranger_kms.cc:326] Created kudu user
I20251026 21:08:26.768487 3772 mini_ranger_kms.cc:343] Created rangerkms user
Oct 26 21:08:26 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
I20251026 21:08:27.163940 3772 mini_ranger_kms.cc:384] Added ranger policy
I20251026 21:08:27.164170 3772 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms:/tmp/dist-test-task7MLemc/build/debug/bin/postgresql.jar:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7MLemc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251026 21:08:27.164193 3772 mini_ranger_kms.cc:240] Using host: 127.3.175.20
I20251026 21:08:27.167502 3772 mini_ranger_kms.cc:276] Ranger KMS PID: 15625
I20251026 21:08:27.167625 3772 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.175.20:37521
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-kms/ews/webapp].
Oct 26, 2025 9:08:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 26, 2025 9:08:28 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-37521"]
Oct 26, 2025 9:08:28 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 26, 2025 9:08:28 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 26, 2025 9:08:29 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 26, 2025 9:08:29 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
I20251026 21:08:29.445760 3772 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.823s user 0.023s sys 0.171s
I20251026 21:08:29.445894 3772 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251026 21:08:29.445989 3772 mini_ranger_kms.cc:401] 127.3.175.20:37521/kms/v1/keys
Oct 26 21:08:29 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761512909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.175.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 26 21:08:29 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.175.20@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
Oct 26 21:08:30 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.175.20@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
Oct 26 21:08:32 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:08:32.846246 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:45299
--webserver_interface=127.3.175.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.175.62:45299
--ranger_config_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.175.20:37521/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:08:32.960639 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:08:32.960930 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:08:32.961022 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:08:32.964673 15678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 21:08:32.964761 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:08:32.964800 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:08:32.964852 15678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 21:08:32.964903 15678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 21:08:32.969851 15678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.175.20:37521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.175.62:45299
--ranger_config_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.175.62:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.175.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:08:32.971124 15678 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:08:32.972049 15678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:08:32.978488 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:08:32.978659 15678 server_base.cc:1047] running on GCE node
W20251026 21:08:32.978488 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:08:32.978488 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:08:32.979481 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:08:32.980510 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:08:32.981685 15678 hybrid_clock.cc:648] HybridClock initialized: now 1761512912981659 us; error 42 us; skew 500 ppm
Oct 26 21:08:32 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512912, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:08:32.984771 15678 init.cc:377] Logged in from keytab as kudu/127.3.175.62@KRBTEST.COM (short username kudu)
I20251026 21:08:32.985991 15678 webserver.cc:492] Webserver started at http://127.3.175.62:34389/ using document root <none> and password file <none>
I20251026 21:08:32.986565 15678 fs_manager.cc:362] Metadata directory not provided
I20251026 21:08:32.986619 15678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:08:32.986814 15678 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 21:08:32 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512912, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.62@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
I20251026 21:08:33.032826 15678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data/instance:
uuid: "55def89864f449c5b3423124a5e2c190"
format_stamp: "Formatted at 2025-10-26 21:08:33 on dist-test-slave-th6x"
server_key: "4d357b5edf413284355ce23536568e2c"
server_key_iv: "601b831d2b3c260bb4517ab6b9858dd0"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:33.033495 15678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal/instance:
uuid: "55def89864f449c5b3423124a5e2c190"
format_stamp: "Formatted at 2025-10-26 21:08:33 on dist-test-slave-th6x"
server_key: "4d357b5edf413284355ce23536568e2c"
server_key_iv: "601b831d2b3c260bb4517ab6b9858dd0"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:33.037257 15678 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 21:08:33.052268 15693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:33.053561 15678 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251026 21:08:33.053699 15678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal
uuid: "55def89864f449c5b3423124a5e2c190"
format_stamp: "Formatted at 2025-10-26 21:08:33 on dist-test-slave-th6x"
server_key: "4d357b5edf413284355ce23536568e2c"
server_key_iv: "601b831d2b3c260bb4517ab6b9858dd0"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:33.053825 15678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:08:33.092545 15678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:08:33.099192 15678 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:08:33.099436 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:08:33.108098 15678 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.62:45299
I20251026 21:08:33.108098 15745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.62:45299 every 8 connection(s)
I20251026 21:08:33.109277 15678 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/data/info.pb
I20251026 21:08:33.112241 15746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 21:08:33.114655 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 15678
I20251026 21:08:33.114805 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/wal/instance
I20251026 21:08:33.115077 3772 external_mini_cluster.cc:1468] Setting key 671f5174f56b18ae1f76c81f1c7ca406
I20251026 21:08:33.118273 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190: Bootstrap starting.
I20251026 21:08:33.120954 15746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190: Neither blocks nor log segments found. Creating new log.
I20251026 21:08:33.121878 15746 log.cc:826] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190: Log is configured to *not* fsync() on all Append() calls
Oct 26 21:08:33 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
I20251026 21:08:33.124555 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190: No bootstrap required, opened a new log
I20251026 21:08:33.128571 15746 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } }
I20251026 21:08:33.128818 15746 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 21:08:33.128924 15746 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55def89864f449c5b3423124a5e2c190, State: Initialized, Role: FOLLOWER
I20251026 21:08:33.129621 15746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } }
I20251026 21:08:33.129807 15746 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 21:08:33.129906 15746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 21:08:33.130008 15746 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 0 FOLLOWER]: Advancing to term 1
I20251026 21:08:33.130700 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:33.116565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.3.175.62:45299)
1026 21:08:33.117060 (+ 495us) server_negotiation.cc:207] Beginning negotiation
1026 21:08:33.117073 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 21:08:33.117116 (+ 43us) server_negotiation.cc:408] Connection header received
1026 21:08:33.117797 (+ 681us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:08:33.117822 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:08:33.118231 (+ 409us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:08:33.118587 (+ 356us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:08:33.119806 (+ 1219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:08:33.120980 (+ 1174us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:08:33.121947 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:08:33.122255 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:08:33.125060 (+ 2805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:08:33.125083 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:08:33.125097 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:08:33.125141 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:08:33.127490 (+ 2349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:08:33.128039 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:08:33.128044 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:08:33.128051 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:08:33.128142 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:08:33.128427 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:08:33.128430 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:08:33.128432 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:08:33.128903 (+ 471us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:08:33.129144 (+ 241us) server_negotiation.cc:1036] Waiting for connection context
1026 21:08:33.129433 (+ 289us) server_negotiation.cc:300] Negotiation successful
1026 21:08:33.129854 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20251026 21:08:33.131139 15746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } }
I20251026 21:08:33.131560 15746 leader_election.cc:304] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55def89864f449c5b3423124a5e2c190; no voters:
I20251026 21:08:33.131911 15746 leader_election.cc:290] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 21:08:33.132016 15751 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 21:08:33.132314 15751 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [term 1 LEADER]: Becoming Leader. State: Replica: 55def89864f449c5b3423124a5e2c190, State: Running, Role: LEADER
I20251026 21:08:33.132705 15751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } }
I20251026 21:08:33.133247 15746 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 21:08:33.134021 15752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55def89864f449c5b3423124a5e2c190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } } }
I20251026 21:08:33.134061 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55def89864f449c5b3423124a5e2c190. Latest consensus state: current_term: 1 leader_uuid: "55def89864f449c5b3423124a5e2c190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def89864f449c5b3423124a5e2c190" member_type: VOTER last_known_addr { host: "127.3.175.62" port: 45299 } } }
I20251026 21:08:33.134483 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [sys.catalog]: This master's current role is: LEADER
I20251026 21:08:33.134771 15752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190 [sys.catalog]: This master's current role is: LEADER
I20251026 21:08:33.135708 15760 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 21:08:33.135855 15746 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251026 21:08:33.142273 15760 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 21:08:33.149477 15770 mvcc.cc:204] Tried to move back new op lower bound from 7215156892253224960 to 7215156892192731136. Current Snapshot: MvccSnapshot[applied={T|T < 7215156892253224960}]
I20251026 21:08:33.151361 15760 catalog_manager.cc:1357] Generated new cluster ID: 540d8614cc7b4e7f90c44d961ec88a99
I20251026 21:08:33.151443 15760 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 21:08:33.175139 15760 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 21:08:33.177172 15760 catalog_manager.cc:1514] Loading token signing keys...
I20251026 21:08:33.184958 15760 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 55def89864f449c5b3423124a5e2c190: Generated new TSK 0
I20251026 21:08:33.186060 15760 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.3.175.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
I20251026 21:08:34.029009 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.175.20:37521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.1:0
--local_ip_for_outbound_sockets=127.3.175.1
--webserver_interface=127.3.175.1
--webserver_port=0
--tserver_master_addrs=127.3.175.62:45299
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 21:08:34.193670 15792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:08:34.193977 15792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:08:34.194067 15792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:08:34.197551 15792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:08:34.197630 15792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:08:34.197744 15792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.1
I20251026 21:08:34.202308 15792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.175.20:37521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.175.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:45299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.1
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:08:34.203536 15792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:08:34.204491 15792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:08:34.211779 15799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:08:34.211710 15802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:08:34.212046 15800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:08:34.212397 15792 server_base.cc:1047] running on GCE node
I20251026 21:08:34.212803 15792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:08:34.213423 15792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:08:34.214577 15792 hybrid_clock.cc:648] HybridClock initialized: now 1761512914214555 us; error 38 us; skew 500 ppm
Oct 26 21:08:34 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:08:34.217633 15792 init.cc:377] Logged in from keytab as kudu/127.3.175.1@KRBTEST.COM (short username kudu)
I20251026 21:08:34.218891 15792 webserver.cc:492] Webserver started at http://127.3.175.1:33717/ using document root <none> and password file <none>
I20251026 21:08:34.219574 15792 fs_manager.cc:362] Metadata directory not provided
I20251026 21:08:34.219633 15792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:08:34.219834 15792 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 21:08:34 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
I20251026 21:08:34.233736 15792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data/instance:
uuid: "1e5742e346a8443eb4070e8c325660ae"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "deda94d52ee45856074f03dac3044824"
server_key_iv: "a73c7e16f43f0db3333449fe85642be6"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.234586 15792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance:
uuid: "1e5742e346a8443eb4070e8c325660ae"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "deda94d52ee45856074f03dac3044824"
server_key_iv: "a73c7e16f43f0db3333449fe85642be6"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.239913 15792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251026 21:08:34.255386 15809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:34.257061 15792 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251026 21:08:34.257328 15792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal
uuid: "1e5742e346a8443eb4070e8c325660ae"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "deda94d52ee45856074f03dac3044824"
server_key_iv: "a73c7e16f43f0db3333449fe85642be6"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.257546 15792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:08:34.300261 15792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:08:34.304352 15792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:08:34.304636 15792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:08:34.305437 15792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:08:34.306823 15792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:08:34.306938 15792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:34.307067 15792 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:08:34.307145 15792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:34.318303 15792 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.1:40533
I20251026 21:08:34.319852 15792 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/data/info.pb
I20251026 21:08:34.321743 15924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.1:40533 every 8 connection(s)
I20251026 21:08:34.328575 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 15792
I20251026 21:08:34.328732 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-0/wal/instance
I20251026 21:08:34.329107 3772 external_mini_cluster.cc:1468] Setting key f4f0beff04ce727c2d6529f0e92e620e
Oct 26 21:08:34 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.1@KRBTEST.COM for kudu/127.3.175.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.175.2@KRBTEST.COM; defaulting to no policy
I20251026 21:08:34.346585 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:34.324208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.175.1:50447 (local address 127.3.175.62:45299)
1026 21:08:34.329834 (+ 5626us) server_negotiation.cc:207] Beginning negotiation
1026 21:08:34.329839 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 21:08:34.329858 (+ 19us) server_negotiation.cc:408] Connection header received
1026 21:08:34.329906 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 21:08:34.329911 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 21:08:34.329986 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 21:08:34.330092 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1026 21:08:34.334252 (+ 4160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:08:34.335109 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 21:08:34.336130 (+ 1021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 21:08:34.336365 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:08:34.341442 (+ 5077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 21:08:34.341474 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 21:08:34.341478 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 21:08:34.341518 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1026 21:08:34.343737 (+ 2219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:08:34.344538 (+ 801us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:08:34.344542 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:08:34.344544 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:08:34.344605 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 21:08:34.345000 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 21:08:34.345006 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 21:08:34.345007 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 21:08:34.345236 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 21:08:34.345366 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 21:08:34.346147 (+ 781us) server_negotiation.cc:300] Negotiation successful
1026 21:08:34.346312 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5517,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.175.2@KRBTEST.COM" created.
I20251026 21:08:34.347405 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:34.325008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:50447)
1026 21:08:34.325581 (+ 573us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:34.325635 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1026 21:08:34.326668 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 21:08:34.330480 (+ 3812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 21:08:34.330495 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 21:08:34.333045 (+ 2550us) client_negotiation.cc:189] Negotiated authn=SASL
1026 21:08:34.334005 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:08:34.334031 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:08:34.335324 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 21:08:34.335328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 21:08:34.335963 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 21:08:34.335978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 21:08:34.336269 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 21:08:34.339134 (+ 2865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 21:08:34.339195 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
1026 21:08:34.341165 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 21:08:34.343955 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:08:34.343963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:08:34.343980 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:08:34.344383 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:08:34.344705 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 21:08:34.344709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 21:08:34.344711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 21:08:34.344847 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 21:08:34.345382 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 21:08:34.345391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 21:08:34.345776 (+ 385us) client_negotiation.cc:769] Sending connection context
1026 21:08:34.346079 (+ 303us) client_negotiation.cc:240] Negotiation successful
1026 21:08:34.346406 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
I20251026 21:08:34.349409 15925 heartbeater.cc:344] Connected to a master server at 127.3.175.62:45299
I20251026 21:08:34.349740 15925 heartbeater.cc:461] Registering TS with master...
I20251026 21:08:34.350515 15925 heartbeater.cc:507] Master 127.3.175.62:45299 requested a full tablet report, sending...
I20251026 21:08:34.352496 15710 ts_manager.cc:194] Registered new tserver with Master: 1e5742e346a8443eb4070e8c325660ae (127.3.175.1:40533)
I20251026 21:08:34.354220 15710 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.175.1@KRBTEST.COM'} at 127.3.175.1:50447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.3.175.62 from keytab /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.3.175.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.175.2@KRBTEST.COM" created.
I20251026 21:08:34.390316 15784 server.cc:273] Received an EOF from the subprocess
F20251026 21:08:34.391346 15769 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.3.175.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab.
@ 0x7fb9e9094dcd google::LogMessage::Fail() at ??:0
I20251026 21:08:34.395001 3772 external_mini_cluster.cc:1366] Running /tmp/dist-test-task7MLemc/build/debug/bin/kudu
/tmp/dist-test-task7MLemc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.175.20:37521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.175.2:0
--local_ip_for_outbound_sockets=127.3.175.2
--webserver_interface=127.3.175.2
--webserver_port=0
--tserver_master_addrs=127.3.175.62:45299
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.3.175.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7fb9e9098b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb9e90947cc google::LogMessage::Flush() at ??:0
@ 0x7fb9e9095f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9ead4a2d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fb9ead470d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fb9ead4b005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561371c0ebea std::function<>::operator()() at ??:0
@ 0x7fb9ea4ee598 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9ead866db start_thread at ??:0
@ 0x7fb9e824b71f clone at ??:0
W20251026 21:08:34.514493 15935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 21:08:34.514751 15935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 21:08:34.514856 15935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 21:08:34.518369 15935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 21:08:34.518453 15935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 21:08:34.518579 15935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.175.2
I20251026 21:08:34.523082 15935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.175.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.175.20:37521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/krb5kdc/kudu.keytab
--principal=kudu/127.3.175.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.175.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.175.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.3.175.62:45299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.175.2
--log_dir=/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 26 Oct 2025 20:43:12 UTC on 24a791456cd2
build id 8730
I20251026 21:08:34.524295 15935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 21:08:34.525210 15935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 21:08:34.532473 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 21:08:34.532473 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:08:34.532738 15935 server_base.cc:1047] running on GCE node
W20251026 21:08:34.532517 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 21:08:34.533313 15935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 21:08:34.533950 15935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 21:08:34.535120 15935 hybrid_clock.cc:648] HybridClock initialized: now 1761512914535098 us; error 48 us; skew 500 ppm
Oct 26 21:08:34 dist-test-slave-th6x krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 21:08:34.538523 15935 init.cc:377] Logged in from keytab as kudu/127.3.175.2@KRBTEST.COM (short username kudu)
I20251026 21:08:34.539860 15935 webserver.cc:492] Webserver started at http://127.3.175.2:43211/ using document root <none> and password file <none>
I20251026 21:08:34.540488 15935 fs_manager.cc:362] Metadata directory not provided
I20251026 21:08:34.540547 15935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 21:08:34.540759 15935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 21:08:34 dist-test-slave-th6x krb5kdc[15207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761512914, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.175.2@KRBTEST.COM for HTTP/127.3.175.20@KRBTEST.COM
I20251026 21:08:34.556207 15935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data/instance:
uuid: "d6f17e07cee640d4b698384325d53d9d"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "e2dbdd915912592051cf3380cc46a9f4"
server_key_iv: "d295ebe20741f4838927ede9b9702ee5"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.556919 15935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance:
uuid: "d6f17e07cee640d4b698384325d53d9d"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "e2dbdd915912592051cf3380cc46a9f4"
server_key_iv: "d295ebe20741f4838927ede9b9702ee5"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.562361 15935 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251026 21:08:34.583427 15960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:34.585023 15935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251026 21:08:34.585284 15935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data,/tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal
uuid: "d6f17e07cee640d4b698384325d53d9d"
format_stamp: "Formatted at 2025-10-26 21:08:34 on dist-test-slave-th6x"
server_key: "e2dbdd915912592051cf3380cc46a9f4"
server_key_iv: "d295ebe20741f4838927ede9b9702ee5"
server_key_version: "kuduclusterkey@0"
I20251026 21:08:34.585520 15935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 21:08:34.610922 15935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 21:08:34.614979 15935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 21:08:34.615259 15935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 21:08:34.616053 15935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 21:08:34.617329 15935 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 21:08:34.617431 15935 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251026 21:08:34.617520 15935 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 21:08:34.617571 15935 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 21:08:34.629663 15935 rpc_server.cc:307] RPC server started. Bound to: 127.3.175.2:45021
I20251026 21:08:34.629824 16073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.175.2:45021 every 8 connection(s)
I20251026 21:08:34.630923 15935 server_base.cc:1179] Dumped server information to /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/data/info.pb
I20251026 21:08:34.633579 3772 external_mini_cluster.cc:1428] Started /tmp/dist-test-task7MLemc/build/debug/bin/kudu as pid 15935
I20251026 21:08:34.633709 3772 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task7MLemc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761512762855912-3772-0/minicluster-data/ts-1/wal/instance
I20251026 21:08:34.633999 3772 external_mini_cluster.cc:1468] Setting key c8f1f7bb7338730a7be519aae66c83de
W20251026 21:08:34.635707 16076 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 21:08:34.633585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:50857)
1026 21:08:34.634897 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:34.635005 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":147,"threads_started":1}
W20251026 21:08:34.636147 16074 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:45299 (0 consecutive failures): Network error: Failed to ping master at 127.3.175.62:45299: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection reset by peer (error 104)
I20251026 21:08:34.636826 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:34.636548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:50907)
1026 21:08:34.636709 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:34.636762 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251026 21:08:34.637547 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:34.637279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:34803)
1026 21:08:34.637425 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:34.637478 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20251026 21:08:34.637691 16074 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 21:08:35.358075 15925 heartbeater.cc:499] Master 127.3.175.62:45299 was elected leader, sending a full tablet report...
I20251026 21:08:35.359403 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:35.358812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:60735)
1026 21:08:35.359183 (+ 371us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:35.359262 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":108,"threads_started":1}
W20251026 21:08:35.359712 15925 heartbeater.cc:646] Failed to heartbeat to 127.3.175.62:45299 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
I20251026 21:08:35.639089 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:35.638410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:39955)
1026 21:08:35.638827 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:35.638940 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20251026 21:08:36.360971 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:36.360494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:40189)
1026 21:08:36.360789 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:36.360847 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20251026 21:08:36.640663 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:36.640180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:39381)
1026 21:08:36.640477 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:36.640540 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20251026 21:08:37.362385 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:37.361857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:49487)
1026 21:08:37.362197 (+ 340us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:37.362274 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20251026 21:08:37.642082 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:37.641606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:42681)
1026 21:08:37.641896 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:37.641976 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20251026 21:08:38.364033 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:38.363511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:46051)
1026 21:08:38.363856 (+ 345us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:38.363930 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":88,"threads_started":1}
I20251026 21:08:38.643481 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:38.642813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:43963)
1026 21:08:38.643193 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:38.643326 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20251026 21:08:39.365399 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:39.364938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.1:34015)
1026 21:08:39.365237 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:39.365301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20251026 21:08:39.644747 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 21:08:39.644325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.175.62:45299 (local address 127.3.175.2:47665)
1026 21:08:39.644588 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1026 21:08:39.644648 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.175.62:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
F20251026 21:08:39.685182 3772 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1761512919 (unix time) try "date -d @1761512919" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000ebc) received by PID 3772 (TID 0x7ffb7ba42d80) from PID 3772; stack trace: ***
@ 0x7ffb876dc980 (unknown) at ??:0
@ 0x7ffb84d14fb7 gsignal at ??:0
@ 0x7ffb84d16921 abort at ??:0
@ 0x7ffb85f82dcd google::LogMessage::Fail() at ??:0
@ 0x7ffb85f86b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffb85f827cc google::LogMessage::Flush() at ??:0
@ 0x7ffb85f83f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffb88d0acab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7ffb88d079c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5606d882892f kudu::SecurityITest::StartCluster() at ??:0
@ 0x5606d8810792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7ffb86f920ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ffb86f86bdb testing::Test::Run() at ??:0
@ 0x7ffb86f86d9d testing::TestInfo::Run() at ??:0
@ 0x7ffb86f87377 testing::TestSuite::Run() at ??:0
@ 0x7ffb86f8777c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7ffb86f9260d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ffb86f86e63 testing::UnitTest::Run() at ??:0
@ 0x5606d8860c7e RUN_ALL_TESTS() at ??:0
@ 0x5606d885fe6b main at ??:0
@ 0x7ffb84cf7bf7 __libc_start_main at ??:0
@ 0x5606d87f806a _start at ??:0
2025-10-26 21:08:39.875 UTC [15656] FATAL: terminating connection due to unexpected postmaster exit
2025-10-26 21:08:39.876 UTC [15552] FATAL: terminating connection due to unexpected postmaster exit
2025-10-26 21:08:39.876 UTC [15553] FATAL: terminating connection due to unexpected postmaster exit
2025-10-26 21:08:39.877 UTC [15550] FATAL: terminating connection due to unexpected postmaster exit
2025-10-26 21:08:39.877 UTC [15549] FATAL: terminating connection due to unexpected postmaster exit
2025-10-26 21:08:39.879 UTC [15548] FATAL: terminating connection due to unexpected postmaster exit