[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:37 dist-test-slave-nb51 krb5kdc[6612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:37 dist-test-slave-nb51 krb5kdc[6612](info): set up 2 sockets
Dec 10 14:57:37 dist-test-slave-nb51 krb5kdc[6612](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251210 14:57:39.730734 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.963s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:57:39 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:57:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:39.860875 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:36557
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:36557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:39.965431 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:39.965672 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:39.965726 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:39.968861 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:57:39.968921 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:39.968947 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:39.968969 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:57:39.968993 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:57:39.973196 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:36557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:36557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:39.974169 6628 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:39.974987 6628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:39.980883 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:39.980926 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:39.980891 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:39.981086 6628 server_base.cc:1047] running on GCE node
I20251210 14:57:39.981709 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:39.982556 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:39.983704 6628 hybrid_clock.cc:648] HybridClock initialized: now 1765378659983696 us; error 25 us; skew 500 ppm
Dec 10 14:57:39 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:39.986096 6628 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:57:39.987130 6628 webserver.cc:492] Webserver started at http://127.6.115.126:35117/ using document root <none> and password file <none>
I20251210 14:57:39.987668 6628 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:39.987717 6628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:39.987891 6628 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:39.989616 6628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "ac2a337b717c46a0b078cc9d2748f75a"
format_stamp: "Formatted at 2025-12-10 14:57:39 on dist-test-slave-nb51"
server_key: "b67cc5a8e4687f643a3b4d1daac3324d"
server_key_iv: "fa83de7f10bed05ed22502b0768f0111"
server_key_version: "encryptionkey@0"
I20251210 14:57:39.990075 6628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "ac2a337b717c46a0b078cc9d2748f75a"
format_stamp: "Formatted at 2025-12-10 14:57:39 on dist-test-slave-nb51"
server_key: "b67cc5a8e4687f643a3b4d1daac3324d"
server_key_iv: "fa83de7f10bed05ed22502b0768f0111"
server_key_version: "encryptionkey@0"
I20251210 14:57:39.993335 6628 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251210 14:57:39.995473 6643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:39.996445 6628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:39.996565 6628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "ac2a337b717c46a0b078cc9d2748f75a"
format_stamp: "Formatted at 2025-12-10 14:57:39 on dist-test-slave-nb51"
server_key: "b67cc5a8e4687f643a3b4d1daac3324d"
server_key_iv: "fa83de7f10bed05ed22502b0768f0111"
server_key_version: "encryptionkey@0"
I20251210 14:57:39.996652 6628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:40.013691 6628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:40.016729 6628 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:40.016916 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:40.024221 6695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:36557 every 8 connection(s)
I20251210 14:57:40.024227 6628 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:36557
I20251210 14:57:40.025260 6628 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:57:40.026055 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 6628
I20251210 14:57:40.026186 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:57:40.027110 6605 external_mini_cluster.cc:1468] Setting key 9c56ef82ce42554e1011673780e91867
I20251210 14:57:40.027987 6696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:40.033478 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a: Bootstrap starting.
I20251210 14:57:40.035517 6696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:40.036188 6696 log.cc:826] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:40.038002 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a: No bootstrap required, opened a new log
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:40.040714 6696 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } }
I20251210 14:57:40.040889 6696 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:40.040954 6696 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2a337b717c46a0b078cc9d2748f75a, State: Initialized, Role: FOLLOWER
I20251210 14:57:40.041424 6696 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } }
I20251210 14:57:40.041688 6696 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:57:40.041767 6696 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:57:40.041863 6696 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:40.042798 6696 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } }
I20251210 14:57:40.043145 6696 leader_election.cc:304] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac2a337b717c46a0b078cc9d2748f75a; no voters:
I20251210 14:57:40.043427 6696 leader_election.cc:290] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:57:40.043534 6701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:40.043726 6701 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [term 1 LEADER]: Becoming Leader. State: Replica: ac2a337b717c46a0b078cc9d2748f75a, State: Running, Role: LEADER
I20251210 14:57:40.044049 6701 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } }
I20251210 14:57:40.044071 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.029215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.6.115.126:36557)
1210 14:57:40.029919 (+ 704us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.029929 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.029969 (+ 40us) server_negotiation.cc:408] Connection header received
1210 14:57:40.031994 (+ 2025us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.032009 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.032275 (+ 266us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.032552 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.034318 (+ 1766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.035051 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.035664 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.035920 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.039051 (+ 3131us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.039068 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.039077 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.039105 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.040817 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.041299 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.041313 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.041316 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.041432 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.041744 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.041755 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.041757 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.042173 (+ 416us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.042367 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.042814 (+ 447us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.043086 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":162,"threads_started":1}
I20251210 14:57:40.044538 6696 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:57:40.045536 6703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac2a337b717c46a0b078cc9d2748f75a. Latest consensus state: current_term: 1 leader_uuid: "ac2a337b717c46a0b078cc9d2748f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } } }
I20251210 14:57:40.045657 6703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:40.045885 6702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac2a337b717c46a0b078cc9d2748f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2a337b717c46a0b078cc9d2748f75a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36557 } } }
I20251210 14:57:40.045953 6702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:40.046490 6710 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:57:40.049055 6710 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:57:40.054032 6710 catalog_manager.cc:1357] Generated new cluster ID: 319ee796cdbe4e84a5cc0bb6761d50ca
I20251210 14:57:40.054101 6710 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:57:40.090453 6710 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:57:40.091351 6710 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:57:40.099450 6710 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ac2a337b717c46a0b078cc9d2748f75a: Generated new TSK 0
I20251210 14:57:40.100279 6710 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:40.146217 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36557
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:40.250577 6724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:40.250871 6724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:40.251032 6724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:40.255646 6724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:40.255714 6724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:40.255797 6724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:57:40.259891 6724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:40.260874 6724 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:40.261701 6724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:40.267795 6732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:40.267789 6729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:40.267786 6730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:40.268244 6724 server_base.cc:1047] running on GCE node
I20251210 14:57:40.268640 6724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:40.269197 6724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:40.270339 6724 hybrid_clock.cc:648] HybridClock initialized: now 1765378660270323 us; error 31 us; skew 500 ppm
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:40.272897 6724 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:57:40.273931 6724 webserver.cc:492] Webserver started at http://127.6.115.65:42249/ using document root <none> and password file <none>
I20251210 14:57:40.274452 6724 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:40.274500 6724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:40.274646 6724 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:40.276300 6724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "b0db27650b29476a8d6b164a9e73d690"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "1c53fbe485a2154c08a5653abb9c6b76"
server_key_iv: "c654d432443231debf942a1dc8a477d6"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.276734 6724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "b0db27650b29476a8d6b164a9e73d690"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "1c53fbe485a2154c08a5653abb9c6b76"
server_key_iv: "c654d432443231debf942a1dc8a477d6"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.280000 6724 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251210 14:57:40.282111 6739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.283140 6724 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:40.283228 6724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "b0db27650b29476a8d6b164a9e73d690"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "1c53fbe485a2154c08a5653abb9c6b76"
server_key_iv: "c654d432443231debf942a1dc8a477d6"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.283301 6724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:40.296734 6724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:40.299432 6724 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:40.299573 6724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:40.300034 6724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:40.300911 6724 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:40.300958 6724 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.300997 6724 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:40.301018 6724 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.310145 6724 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:40831
I20251210 14:57:40.310185 6852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:40831 every 8 connection(s)
I20251210 14:57:40.311086 6724 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:57:40.311517 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 6724
I20251210 14:57:40.311637 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:57:40.311883 6605 external_mini_cluster.cc:1468] Setting key 3679d1ceaf883f66228f4f1091b6415c
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:57:40.325047 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.312859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:47517 (local address 127.6.115.126:36557)
1210 14:57:40.313010 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.313014 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.313785 (+ 771us) server_negotiation.cc:408] Connection header received
1210 14:57:40.314662 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.314666 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.314727 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.314812 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.316792 (+ 1980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.317580 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.318538 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.318684 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.321416 (+ 2732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.321433 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.321436 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.321462 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.323037 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.323496 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.323500 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.323502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.323572 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.323832 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.323834 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.323835 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.324027 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.324122 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.324736 (+ 614us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.324882 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:57:40.325800 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.313152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36557 (local address 127.6.115.65:47517)
1210 14:57:40.313655 (+ 503us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:40.313692 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:40.314426 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:40.314978 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:40.314986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:40.315396 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:40.316638 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.316659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.317720 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.317726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:40.318399 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.318414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.318677 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.319356 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:40.319400 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:40.321235 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:40.323145 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.323150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.323159 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.323404 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.323666 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.323668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.323669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.323756 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.324138 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:40.324146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:40.324459 (+ 313us) client_negotiation.cc:769] Sending connection context
1210 14:57:40.324682 (+ 223us) client_negotiation.cc:240] Negotiation successful
1210 14:57:40.324961 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":166,"threads_started":1}
I20251210 14:57:40.327133 6853 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36557
I20251210 14:57:40.327360 6853 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:40.327836 6853 heartbeater.cc:507] Master 127.6.115.126:36557 requested a full tablet report, sending...
I20251210 14:57:40.332758 6660 ts_manager.cc:194] Registered new tserver with Master: b0db27650b29476a8d6b164a9e73d690 (127.6.115.65:40831)
I20251210 14:57:40.334604 6660 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:47517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:40.362430 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36557
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:40.468204 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:40.468413 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:40.468472 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:40.471632 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:40.471693 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:40.471773 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:57:40.476220 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:40.477334 6860 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:40.478185 6860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:40.484361 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:40.484354 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:40.484354 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:40.484566 6860 server_base.cc:1047] running on GCE node
I20251210 14:57:40.485029 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:40.485603 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:40.486743 6860 hybrid_clock.cc:648] HybridClock initialized: now 1765378660486735 us; error 28 us; skew 500 ppm
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:40.489199 6860 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:57:40.490234 6860 webserver.cc:492] Webserver started at http://127.6.115.66:44359/ using document root <none> and password file <none>
I20251210 14:57:40.490751 6860 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:40.490803 6860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:40.490986 6860 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:40.492599 6860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "2e373b0ea12342abaad60a8bc83aef22"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "82332918309d05c577e873c8fe809a54"
server_key_iv: "2c382d5fe167a2d354d010f5dbf89956"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.493031 6860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "2e373b0ea12342abaad60a8bc83aef22"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "82332918309d05c577e873c8fe809a54"
server_key_iv: "2c382d5fe167a2d354d010f5dbf89956"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.496390 6860 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:40.498448 6875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.499400 6860 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:57:40.499497 6860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "2e373b0ea12342abaad60a8bc83aef22"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "82332918309d05c577e873c8fe809a54"
server_key_iv: "2c382d5fe167a2d354d010f5dbf89956"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.499574 6860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:40.509264 6860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:40.511673 6860 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:40.511826 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:40.512290 6860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:40.513090 6860 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:40.513149 6860 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.513204 6860 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:40.513233 6860 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.522073 6860 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:39769
I20251210 14:57:40.522109 6988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:39769 every 8 connection(s)
I20251210 14:57:40.522933 6860 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:57:40.527642 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 6860
I20251210 14:57:40.527738 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:57:40.527987 6605 external_mini_cluster.cc:1468] Setting key a81903321ab72fef5dc259e2d4aab07e
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:40.535398 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.524518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:52253 (local address 127.6.115.126:36557)
1210 14:57:40.524645 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.524649 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.525262 (+ 613us) server_negotiation.cc:408] Connection header received
1210 14:57:40.526148 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.526151 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.526199 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.526301 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.527876 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.528408 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.529135 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.529413 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.531726 (+ 2313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.531741 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.531743 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.531766 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.533255 (+ 1489us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.533950 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.533952 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.533954 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.533995 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.534335 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.534337 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.534339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.534469 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.534549 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.535162 (+ 613us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.535273 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:57:40.536350 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.524743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36557 (local address 127.6.115.66:52253)
1210 14:57:40.525125 (+ 382us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:40.525159 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:40.525925 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:40.526431 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:40.526438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:40.526808 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:40.527728 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.527748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.528515 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.528519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:40.529020 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.529029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.529242 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.529866 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:40.529891 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:40.531560 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:40.533429 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.533439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.533453 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.533824 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.534081 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.534087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.534091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.534242 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.534577 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:40.534586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:40.534891 (+ 305us) client_negotiation.cc:769] Sending connection context
1210 14:57:40.535139 (+ 248us) client_negotiation.cc:240] Negotiation successful
1210 14:57:40.535465 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20251210 14:57:40.537349 6989 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36557
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:57:40.537658 6989 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:40.538216 6989 heartbeater.cc:507] Master 127.6.115.126:36557 requested a full tablet report, sending...
I20251210 14:57:40.539113 6660 ts_manager.cc:194] Registered new tserver with Master: 2e373b0ea12342abaad60a8bc83aef22 (127.6.115.66:39769)
I20251210 14:57:40.539701 6660 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:52253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:40.572217 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36557
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:40.673064 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:40.673260 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:40.673317 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:40.676586 6996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:40.676642 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:40.676736 6996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:57:40.680765 6996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:40.681850 6996 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:40.682683 6996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:40.688959 7001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:40.688998 7002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:40.689087 6996 server_base.cc:1047] running on GCE node
W20251210 14:57:40.688989 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:40.689616 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:40.690136 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:40.691286 6996 hybrid_clock.cc:648] HybridClock initialized: now 1765378660691274 us; error 32 us; skew 500 ppm
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:40.693785 6996 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:57:40.694761 6996 webserver.cc:492] Webserver started at http://127.6.115.67:45081/ using document root <none> and password file <none>
I20251210 14:57:40.695286 6996 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:40.695336 6996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:40.695513 6996 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:40.697120 6996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "595825b78c3b43cb8ae846ff1def9054"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "7f6a8de5d5b42bb83d3259361ba79c61"
server_key_iv: "b22eb5c6225fdad2fc316f03e93bd9bb"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.697605 6996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "595825b78c3b43cb8ae846ff1def9054"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "7f6a8de5d5b42bb83d3259361ba79c61"
server_key_iv: "b22eb5c6225fdad2fc316f03e93bd9bb"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.700795 6996 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251210 14:57:40.702955 7011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.704011 6996 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251210 14:57:40.704098 6996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "595825b78c3b43cb8ae846ff1def9054"
format_stamp: "Formatted at 2025-12-10 14:57:40 on dist-test-slave-nb51"
server_key: "7f6a8de5d5b42bb83d3259361ba79c61"
server_key_iv: "b22eb5c6225fdad2fc316f03e93bd9bb"
server_key_version: "encryptionkey@0"
I20251210 14:57:40.704205 6996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:40.723202 6996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:40.726091 6996 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:40.726241 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:40.726711 6996 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:40.727524 6996 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:40.727567 6996 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.727603 6996 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:40.727625 6996 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:40.736083 6996 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:38629
I20251210 14:57:40.736133 7124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:38629 every 8 connection(s)
I20251210 14:57:40.736943 6996 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:57:40.737210 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 6996
I20251210 14:57:40.737318 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:57:40.737622 6605 external_mini_cluster.cc:1468] Setting key 5540a7cfff9e01921718731c318db64b
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:40.749979 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.740216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:55861 (local address 127.6.115.126:36557)
1210 14:57:40.740388 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.740395 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.740761 (+ 366us) server_negotiation.cc:408] Connection header received
1210 14:57:40.741629 (+ 868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.741632 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.741682 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.741794 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.743407 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.743907 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.744537 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.744668 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.746861 (+ 2193us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.746878 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.746882 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.746907 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.748325 (+ 1418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.748808 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.748811 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.748812 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.748862 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.749094 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.749096 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.749097 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.749233 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.749306 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.749754 (+ 448us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.749853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251210 14:57:40.750595 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.739164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36557 (local address 127.6.115.67:55861)
1210 14:57:40.740615 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:40.740661 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:40.741424 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:40.741948 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:40.741956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:40.742337 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:40.743246 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.743261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.744016 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.744019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:40.744438 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:40.744445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.744601 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.745162 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:40.745181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:40.746696 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:40.748450 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.748456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.748466 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.748684 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.748945 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:40.748947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:40.748948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:40.749027 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:40.749305 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:40.749310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:40.749559 (+ 249us) client_negotiation.cc:769] Sending connection context
1210 14:57:40.749724 (+ 165us) client_negotiation.cc:240] Negotiation successful
1210 14:57:40.749931 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":133,"threads_started":1}
I20251210 14:57:40.751564 7125 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36557
I20251210 14:57:40.751789 7125 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:40.752243 7125 heartbeater.cc:507] Master 127.6.115.126:36557 requested a full tablet report, sending...
I20251210 14:57:40.753091 6660 ts_manager.cc:194] Registered new tserver with Master: 595825b78c3b43cb8ae846ff1def9054 (127.6.115.67:38629)
I20251210 14:57:40.753677 6660 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:55861
I20251210 14:57:40.763485 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:40.785807 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.778307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.6.115.65:40831)
1210 14:57:40.778601 (+ 294us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.778607 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.778631 (+ 24us) server_negotiation.cc:408] Connection header received
1210 14:57:40.778686 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.778690 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.778849 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.778959 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.779679 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.780368 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.780903 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.781088 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.783054 (+ 1966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.783074 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.783084 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.783108 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.784514 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.784836 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.784840 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.784841 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.784881 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.785015 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.785017 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.785018 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.785163 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.785291 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.785487 (+ 196us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.785597 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20251210 14:57:40.786613 6767 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58254
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[6612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:40.808604 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:40.801686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.6.115.65:40831)
1210 14:57:40.801824 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:40.801828 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:40.801872 (+ 44us) server_negotiation.cc:408] Connection header received
1210 14:57:40.801974 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:40.801976 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:40.802022 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:40.802097 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:40.802817 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.803378 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:40.803971 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:40.804138 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:40.806112 (+ 1974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:40.806126 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:40.806129 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:40.806151 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:40.807424 (+ 1273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.807733 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.807735 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.807737 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.807770 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:40.807903 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:40.807905 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:40.807906 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:40.808005 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:40.808108 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:40.808311 (+ 203us) server_negotiation.cc:300] Negotiation successful
1210 14:57:40.808460 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251210 14:57:40.809449 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 6724
I20251210 14:57:40.815142 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 6860
I20251210 14:57:40.822137 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 6996
I20251210 14:57:40.827278 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 6628
2025-12-10T14:57:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3087 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[7139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[7139](info): set up 2 sockets
Dec 10 14:57:40 dist-test-slave-nb51 krb5kdc[7139](info): commencing operation
krb5kdc: starting...
W20251210 14:57:42.797904 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.949s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:57:42 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:42.924659 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:35449
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:43.024632 7155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:43.024849 7155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:43.024902 7155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:43.028064 7155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:57:43.028116 7155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:43.028138 7155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:43.028156 7155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:57:43.028173 7155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:57:43.032119 7155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:35449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:43.032987 7155 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:43.033747 7155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:43.038827 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.038872 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.038926 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:43.039004 7155 server_base.cc:1047] running on GCE node
I20251210 14:57:43.039574 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:43.040437 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:43.041581 7155 hybrid_clock.cc:648] HybridClock initialized: now 1765378663041550 us; error 49 us; skew 500 ppm
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:43.044137 7155 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:57:43.045114 7155 webserver.cc:492] Webserver started at http://127.6.115.126:34881/ using document root <none> and password file <none>
I20251210 14:57:43.045648 7155 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:43.045701 7155 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:43.045861 7155 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:43.047451 7155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "0c99d9fe512a4fb6b55a1a244a31a97e"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "b06474c49714ff4bdfd699ccdc5c9348"
server_key_iv: "e2af18b39d5671d90a536ce26f8e6a9f"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.047920 7155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "0c99d9fe512a4fb6b55a1a244a31a97e"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "b06474c49714ff4bdfd699ccdc5c9348"
server_key_iv: "e2af18b39d5671d90a536ce26f8e6a9f"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.051216 7155 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:43.053418 7170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.054349 7155 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:43.054438 7155 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "0c99d9fe512a4fb6b55a1a244a31a97e"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "b06474c49714ff4bdfd699ccdc5c9348"
server_key_iv: "e2af18b39d5671d90a536ce26f8e6a9f"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.054507 7155 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:43.070657 7155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:43.076037 7155 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:43.076197 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:43.083065 7155 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:35449
I20251210 14:57:43.083065 7222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:35449 every 8 connection(s)
I20251210 14:57:43.084018 7155 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:57:43.086592 7223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:43.089932 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7155
I20251210 14:57:43.090067 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:57:43.090309 6605 external_mini_cluster.cc:1468] Setting key 9a4e5eeebd3ed561f5fcb3e6f676b962
I20251210 14:57:43.092329 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e: Bootstrap starting.
I20251210 14:57:43.094358 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:43.095013 7223 log.cc:826] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:43.096688 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e: No bootstrap required, opened a new log
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:43.099845 7223 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } }
I20251210 14:57:43.100078 7223 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:43.100152 7223 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c99d9fe512a4fb6b55a1a244a31a97e, State: Initialized, Role: FOLLOWER
I20251210 14:57:43.100579 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } }
I20251210 14:57:43.100726 7223 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:57:43.100819 7223 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:57:43.100911 7223 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:43.101728 7223 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } }
I20251210 14:57:43.102072 7223 leader_election.cc:304] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c99d9fe512a4fb6b55a1a244a31a97e; no voters:
I20251210 14:57:43.102370 7223 leader_election.cc:290] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:57:43.102445 7228 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:43.102630 7228 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [term 1 LEADER]: Becoming Leader. State: Replica: 0c99d9fe512a4fb6b55a1a244a31a97e, State: Running, Role: LEADER
I20251210 14:57:43.102941 7228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } }
I20251210 14:57:43.103180 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.091562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.6.115.126:35449)
1210 14:57:43.091943 (+ 381us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.091952 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.091992 (+ 40us) server_negotiation.cc:408] Connection header received
1210 14:57:43.092527 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.092545 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.092819 (+ 274us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.093095 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.093986 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.094746 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.095395 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.095679 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.097863 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.097883 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.097898 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.097941 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.100091 (+ 2150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.100579 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.100585 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.100593 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.100688 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.100931 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.100936 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.100938 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.101306 (+ 368us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.102114 (+ 808us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.102276 (+ 162us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.102506 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20251210 14:57:43.103466 7223 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:57:43.104002 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c99d9fe512a4fb6b55a1a244a31a97e. Latest consensus state: current_term: 1 leader_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } } }
I20251210 14:57:43.104122 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:43.105481 7233 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:57:43.105966 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c99d9fe512a4fb6b55a1a244a31a97e" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35449 } } }
I20251210 14:57:43.106065 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:43.107841 7233 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:57:43.112334 7233 catalog_manager.cc:1357] Generated new cluster ID: 4facb0b2a23a477bae638970b8425e60
I20251210 14:57:43.112411 7233 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:57:43.123659 7233 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:57:43.124843 7233 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:57:43.134008 7233 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0c99d9fe512a4fb6b55a1a244a31a97e: Generated new TSK 0
I20251210 14:57:43.134816 7233 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:43.188301 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35449
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:43.289247 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:43.289471 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:43.289527 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:43.292718 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:43.292773 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:43.292847 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:57:43.296777 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:43.297698 7251 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:43.298435 7251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:43.304519 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.304533 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.304523 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:43.304684 7251 server_base.cc:1047] running on GCE node
I20251210 14:57:43.305117 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:43.305675 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:43.306821 7251 hybrid_clock.cc:648] HybridClock initialized: now 1765378663306803 us; error 42 us; skew 500 ppm
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:43.309213 7251 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:57:43.310140 7251 webserver.cc:492] Webserver started at http://127.6.115.65:34493/ using document root <none> and password file <none>
I20251210 14:57:43.310626 7251 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:43.310669 7251 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:43.310817 7251 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:43.312345 7251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "03e1cdbdab75416da175debd4dfebcc5"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "65470d75c0c150ede2a18dcb2a0557e2"
server_key_iv: "7437b3116a56d91f38a982abb47f86e8"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.312781 7251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "03e1cdbdab75416da175debd4dfebcc5"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "65470d75c0c150ede2a18dcb2a0557e2"
server_key_iv: "7437b3116a56d91f38a982abb47f86e8"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.315923 7251 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251210 14:57:43.317909 7266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.318836 7251 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:43.318938 7251 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "03e1cdbdab75416da175debd4dfebcc5"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "65470d75c0c150ede2a18dcb2a0557e2"
server_key_iv: "7437b3116a56d91f38a982abb47f86e8"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.319017 7251 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:43.328573 7251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:43.331247 7251 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:43.331418 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:43.331923 7251 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:43.332701 7251 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:43.332765 7251 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.332818 7251 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:43.332849 7251 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.341708 7251 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:40609
I20251210 14:57:43.341733 7379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:40609 every 8 connection(s)
I20251210 14:57:43.342628 7251 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:57:43.343068 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7251
I20251210 14:57:43.343194 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:57:43.343442 6605 external_mini_cluster.cc:1468] Setting key 4f6d275feaeb7ac7c88ba7e1002f7dc8
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:57:43.354996 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.344309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:51005 (local address 127.6.115.126:35449)
1210 14:57:43.344462 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.344466 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.345201 (+ 735us) server_negotiation.cc:408] Connection header received
1210 14:57:43.346044 (+ 843us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.346048 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.346100 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.346171 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.347812 (+ 1641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.348309 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.348943 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.349154 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.351217 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.351233 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.351236 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.351261 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.352857 (+ 1596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.353383 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.353389 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.353393 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.353442 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.353792 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.353798 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.353803 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.354009 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.354076 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.354609 (+ 533us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.354760 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:57:43.355445 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.344603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35449 (local address 127.6.115.65:51005)
1210 14:57:43.345074 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.345114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.345865 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.346338 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.346346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.346721 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:43.347672 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.347687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.348423 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.348427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.348836 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.348844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.349050 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.349703 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:43.349725 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:43.351069 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:43.352986 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.352991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.353000 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.353265 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.353530 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.353539 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.353541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.353666 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.354158 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:43.354163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:43.354366 (+ 203us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.354530 (+ 164us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.354730 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20251210 14:57:43.356619 7380 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35449
I20251210 14:57:43.356846 7380 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:43.357415 7380 heartbeater.cc:507] Master 127.6.115.126:35449 requested a full tablet report, sending...
I20251210 14:57:43.359015 7187 ts_manager.cc:194] Registered new tserver with Master: 03e1cdbdab75416da175debd4dfebcc5 (127.6.115.65:40609)
I20251210 14:57:43.360670 7187 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:51005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:43.388038 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35449
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:43.491043 7387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:43.491271 7387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:43.491366 7387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:43.494611 7387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:43.494673 7387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:43.494783 7387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:57:43.498873 7387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:43.499917 7387 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:43.500717 7387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:43.506894 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.506912 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.506897 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:43.507289 7387 server_base.cc:1047] running on GCE node
I20251210 14:57:43.507630 7387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:43.508140 7387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:43.509274 7387 hybrid_clock.cc:648] HybridClock initialized: now 1765378663509257 us; error 33 us; skew 500 ppm
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:43.511813 7387 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:57:43.512771 7387 webserver.cc:492] Webserver started at http://127.6.115.66:42645/ using document root <none> and password file <none>
I20251210 14:57:43.513299 7387 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:43.513351 7387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:43.513545 7387 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:43.515138 7387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "be79ac2043204dadb558b3ae13a380ae"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "4bda3acd0c33e51934f5da525750d05b"
server_key_iv: "fc7c6654cdead951645423cbc8413da4"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.515607 7387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "be79ac2043204dadb558b3ae13a380ae"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "4bda3acd0c33e51934f5da525750d05b"
server_key_iv: "fc7c6654cdead951645423cbc8413da4"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.518896 7387 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251210 14:57:43.520965 7402 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.521973 7387 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:43.522069 7387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "be79ac2043204dadb558b3ae13a380ae"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "4bda3acd0c33e51934f5da525750d05b"
server_key_iv: "fc7c6654cdead951645423cbc8413da4"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.522153 7387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:43.549849 7387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:43.552989 7387 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:43.553236 7387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:43.553841 7387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:43.554795 7387 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:43.554844 7387 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.554908 7387 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:43.554934 7387 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.563997 7387 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:35425
I20251210 14:57:43.564013 7515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:35425 every 8 connection(s)
I20251210 14:57:43.564909 7387 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:43.573276 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7387
I20251210 14:57:43.573416 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:57:43.573642 6605 external_mini_cluster.cc:1468] Setting key 61f010e72619cf331edff0787d7afa71
I20251210 14:57:43.576438 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.566465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:34369 (local address 127.6.115.126:35449)
1210 14:57:43.566583 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.566586 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.567283 (+ 697us) server_negotiation.cc:408] Connection header received
1210 14:57:43.568083 (+ 800us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.568087 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.568138 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.568216 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.569880 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.570372 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.570947 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.571068 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.573170 (+ 2102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.573190 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.573192 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.573216 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.574670 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.575173 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.575176 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.575177 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.575217 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.575472 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.575475 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.575477 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.575610 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.575702 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.576166 (+ 464us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.576283 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251210 14:57:43.577019 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.566682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35449 (local address 127.6.115.66:34369)
1210 14:57:43.567137 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.567172 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.567881 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.568369 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.568377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.568758 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:43.569725 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.569740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.570456 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.570458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.570867 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.570873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.571026 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.571612 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:43.571634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:43.573035 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:43.574799 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.574806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.574816 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.575077 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.575285 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.575287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.575289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.575375 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.575706 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:43.575715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:43.575949 (+ 234us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.576124 (+ 175us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.576372 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20251210 14:57:43.578120 7516 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35449
I20251210 14:57:43.578353 7516 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:43.578809 7516 heartbeater.cc:507] Master 127.6.115.126:35449 requested a full tablet report, sending...
I20251210 14:57:43.579741 7187 ts_manager.cc:194] Registered new tserver with Master: be79ac2043204dadb558b3ae13a380ae (127.6.115.66:35425)
I20251210 14:57:43.580277 7187 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:34369
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:43.616456 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35449
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:43.721424 7523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:43.721635 7523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:43.721700 7523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:43.724967 7523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:43.725033 7523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:43.725131 7523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:57:43.729275 7523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:43.730340 7523 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:43.731194 7523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:43.737746 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.737763 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:43.737776 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:43.737946 7523 server_base.cc:1047] running on GCE node
I20251210 14:57:43.738293 7523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:43.738838 7523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:43.740000 7523 hybrid_clock.cc:648] HybridClock initialized: now 1765378663739982 us; error 37 us; skew 500 ppm
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:43.742614 7523 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:57:43.743628 7523 webserver.cc:492] Webserver started at http://127.6.115.67:45727/ using document root <none> and password file <none>
I20251210 14:57:43.744144 7523 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:43.744208 7523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:43.744361 7523 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:43.746054 7523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "d7b0e6fea63245dda5d20db8f5bec3e6"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "0591d1334d91ae6131f5335df9b76623"
server_key_iv: "631541b22767e72004e3702651885e56"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.746480 7523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b0e6fea63245dda5d20db8f5bec3e6"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "0591d1334d91ae6131f5335df9b76623"
server_key_iv: "631541b22767e72004e3702651885e56"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.749806 7523 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:57:43.752370 7538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.753325 7523 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:57:43.753453 7523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "d7b0e6fea63245dda5d20db8f5bec3e6"
format_stamp: "Formatted at 2025-12-10 14:57:43 on dist-test-slave-nb51"
server_key: "0591d1334d91ae6131f5335df9b76623"
server_key_iv: "631541b22767e72004e3702651885e56"
server_key_version: "encryptionkey@0"
I20251210 14:57:43.753543 7523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:43.762666 7523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:43.764991 7523 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:43.765139 7523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:43.765661 7523 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:43.766431 7523 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:43.766474 7523 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.766515 7523 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:43.766546 7523 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:43.774983 7523 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:43701
I20251210 14:57:43.775010 7651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:43701 every 8 connection(s)
I20251210 14:57:43.775807 7523 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:57:43.781603 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7523
I20251210 14:57:43.781718 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:57:43.781965 6605 external_mini_cluster.cc:1468] Setting key 2fbbfb1967bb844b1bdf1977d39d4c09
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:43.786863 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.777308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:55703 (local address 127.6.115.126:35449)
1210 14:57:43.777494 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.777500 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.778084 (+ 584us) server_negotiation.cc:408] Connection header received
1210 14:57:43.778817 (+ 733us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.778820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.778865 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.778941 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.780449 (+ 1508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.780954 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.781583 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.781812 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.783914 (+ 2102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.783929 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.783931 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.783954 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.785234 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.785825 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.785829 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.785831 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.785874 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.786075 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.786077 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.786078 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.786202 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.786270 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.786649 (+ 379us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.786752 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20251210 14:57:43.787473 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.777573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35449 (local address 127.6.115.67:55703)
1210 14:57:43.777960 (+ 387us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.777991 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.778657 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.779077 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.779086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.779431 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:43.780325 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.780337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.781046 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.781049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.781477 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.781484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.781642 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.782216 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:43.782254 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:43.783804 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:43.785353 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.785382 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.785396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.785736 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.785935 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:43.785937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:43.785938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:43.786013 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:43.786276 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:43.786281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:43.786468 (+ 187us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.786616 (+ 148us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.786804 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
I20251210 14:57:43.788391 7652 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35449
I20251210 14:57:43.788592 7652 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:43.789022 7652 heartbeater.cc:507] Master 127.6.115.126:35449 requested a full tablet report, sending...
I20251210 14:57:43.789877 7187 ts_manager.cc:194] Registered new tserver with Master: d7b0e6fea63245dda5d20db8f5bec3e6 (127.6.115.67:43701)
I20251210 14:57:43.790387 7187 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:55703
I20251210 14:57:43.795198 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:57:43.804098 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.797906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.6.115.126:35449)
1210 14:57:43.798039 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.798044 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.798159 (+ 115us) server_negotiation.cc:408] Connection header received
1210 14:57:43.798272 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.798276 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.798317 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.798376 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.799141 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.799614 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.800269 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.800470 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.801474 (+ 1004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.801489 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.801492 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.801514 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.802793 (+ 1279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.803146 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.803148 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.803150 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.803188 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.803379 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.803382 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.803384 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.803525 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.803638 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.803823 (+ 185us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.803953 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:57:43.807775 7187 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:43.810037 7187 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:43.821918 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.817631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.6.115.67:43701)
1210 14:57:43.818150 (+ 519us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.818158 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.818184 (+ 26us) server_negotiation.cc:408] Connection header received
1210 14:57:43.818419 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.818425 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.818619 (+ 194us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.818758 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:43.819383 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.820198 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.821012 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.821555 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.821654 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.821712 (+ 58us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.821785 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":116,"threads_started":1}
I20251210 14:57:43.821976 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.817524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43701 (local address 127.0.0.1:43954)
1210 14:57:43.818076 (+ 552us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.818113 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.818234 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.818919 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.818925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.818944 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:43.819252 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.819266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.820299 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.820303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.820865 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.820875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.820996 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.821043 (+ 47us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.821118 (+ 75us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.821193 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":78,"threads_started":1}
I20251210 14:57:43.822013 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.817760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:35425 (local address 127.0.0.1:56372)
1210 14:57:43.818317 (+ 557us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.818334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.818426 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.819304 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.819310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.819334 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:43.819611 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.819619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.820932 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.820936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.821702 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.821713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.821802 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.821818 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.821865 (+ 47us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.821929 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":114,"threads_started":1}
I20251210 14:57:43.822331 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.818172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:40609 (local address 127.0.0.1:53936)
1210 14:57:43.818762 (+ 590us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.818776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.818850 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.819491 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.819495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.819512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:43.819704 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.819714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.821177 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.821180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.822000 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.822011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.822111 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.822126 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.822174 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.822224 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":101,"threads_started":1}
I20251210 14:57:43.823276 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.818241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.6.115.66:35425)
1210 14:57:43.818878 (+ 637us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.818900 (+ 22us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.818924 (+ 24us) server_negotiation.cc:408] Connection header received
1210 14:57:43.819004 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.819008 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.819171 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.819299 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:43.819731 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.820806 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.822452 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.822923 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.823015 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.823076 (+ 61us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.823157 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":121,"threads_started":1}
I20251210 14:57:43.823503 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.818552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.6.115.65:40609)
1210 14:57:43.819087 (+ 535us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.819093 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.819112 (+ 19us) server_negotiation.cc:408] Connection header received
1210 14:57:43.819184 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.819189 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.819328 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.819860 (+ 532us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:43.819995 (+ 135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.821053 (+ 1058us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.822663 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.823174 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.823248 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.823298 (+ 50us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.823375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":70,"threads_started":1}
I20251210 14:57:43.825158 7586 tablet_service.cc:1505] Processing CreateTablet for tablet e199afd7232542e6b74a8bee9e7687fc (DEFAULT_TABLE table=test-table [id=0c459bff345049a3972501e730a4a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:43.825795 7450 tablet_service.cc:1505] Processing CreateTablet for tablet e199afd7232542e6b74a8bee9e7687fc (DEFAULT_TABLE table=test-table [id=0c459bff345049a3972501e730a4a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:43.826264 7586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e199afd7232542e6b74a8bee9e7687fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:43.825949 7314 tablet_service.cc:1505] Processing CreateTablet for tablet e199afd7232542e6b74a8bee9e7687fc (DEFAULT_TABLE table=test-table [id=0c459bff345049a3972501e730a4a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:43.826607 7450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e199afd7232542e6b74a8bee9e7687fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:43.826823 7314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e199afd7232542e6b74a8bee9e7687fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:43.831707 7669 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: Bootstrap starting.
I20251210 14:57:43.832783 7670 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Bootstrap starting.
I20251210 14:57:43.834015 7669 tablet_bootstrap.cc:654] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:43.834450 7670 tablet_bootstrap.cc:654] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:43.834676 7669 log.cc:826] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:43.835069 7670 log.cc:826] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:43.835234 7671 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Bootstrap starting.
I20251210 14:57:43.836309 7669 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: No bootstrap required, opened a new log
I20251210 14:57:43.836486 7669 ts_tablet_manager.cc:1403] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251210 14:57:43.836735 7670 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: No bootstrap required, opened a new log
I20251210 14:57:43.836830 7671 tablet_bootstrap.cc:654] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:43.836900 7670 ts_tablet_manager.cc:1403] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251210 14:57:43.837464 7671 log.cc:826] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:43.838938 7671 tablet_bootstrap.cc:492] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: No bootstrap required, opened a new log
I20251210 14:57:43.839097 7671 ts_tablet_manager.cc:1403] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251210 14:57:43.840389 7669 raft_consensus.cc:359] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.840597 7669 raft_consensus.cc:385] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:43.840662 7669 raft_consensus.cc:740] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be79ac2043204dadb558b3ae13a380ae, State: Initialized, Role: FOLLOWER
I20251210 14:57:43.840678 7670 raft_consensus.cc:359] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.840858 7670 raft_consensus.cc:385] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:43.840926 7670 raft_consensus.cc:740] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e1cdbdab75416da175debd4dfebcc5, State: Initialized, Role: FOLLOWER
I20251210 14:57:43.841166 7669 consensus_queue.cc:260] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.841295 7670 consensus_queue.cc:260] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.841934 7380 heartbeater.cc:499] Master 127.6.115.126:35449 was elected leader, sending a full tablet report...
I20251210 14:57:43.842044 7670 ts_tablet_manager.cc:1434] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251210 14:57:43.842093 7671 raft_consensus.cc:359] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.842262 7671 raft_consensus.cc:385] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:43.842314 7671 raft_consensus.cc:740] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b0e6fea63245dda5d20db8f5bec3e6, State: Initialized, Role: FOLLOWER
I20251210 14:57:43.842550 7516 heartbeater.cc:499] Master 127.6.115.126:35449 was elected leader, sending a full tablet report...
I20251210 14:57:43.842705 7669 ts_tablet_manager.cc:1434] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251210 14:57:43.842734 7671 consensus_queue.cc:260] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.843258 7652 heartbeater.cc:499] Master 127.6.115.126:35449 was elected leader, sending a full tablet report...
I20251210 14:57:43.843458 7671 ts_tablet_manager.cc:1434] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20251210 14:57:43.843849 7381 tablet.cc:2378] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:43.931402 7675 raft_consensus.cc:493] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:43.931598 7675 raft_consensus.cc:515] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.932724 7675 leader_election.cc:290] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b0e6fea63245dda5d20db8f5bec3e6 (127.6.115.67:43701), 03e1cdbdab75416da175debd4dfebcc5 (127.6.115.65:40609)
I20251210 14:57:43.936008 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.932956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43701 (local address 127.6.115.66:45933)
1210 14:57:43.933089 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.933106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.933201 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.933537 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.933542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.933568 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:43.933818 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.933825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.934791 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.934793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.935621 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.935631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.935737 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.935756 (+ 19us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.935801 (+ 45us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.935862 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251210 14:57:43.936460 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.933073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:45933 (local address 127.6.115.67:43701)
1210 14:57:43.933218 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.933224 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.933254 (+ 30us) server_negotiation.cc:408] Connection header received
1210 14:57:43.933318 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.933323 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.933416 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.933523 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:43.933954 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.934670 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.935738 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.936241 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.936277 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.936325 (+ 48us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.936378 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:57:43.936856 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.933351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:38555 (local address 127.6.115.65:40609)
1210 14:57:43.933539 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.933544 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.933606 (+ 62us) server_negotiation.cc:408] Connection header received
1210 14:57:43.933760 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.933764 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.933808 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.933911 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:43.934276 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.934959 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.935987 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.936482 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.936515 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.936711 (+ 196us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.936765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251210 14:57:43.937018 7606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e199afd7232542e6b74a8bee9e7687fc" candidate_uuid: "be79ac2043204dadb558b3ae13a380ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" is_pre_election: true
I20251210 14:57:43.937139 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.933224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:40609 (local address 127.6.115.66:38555)
1210 14:57:43.933537 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:43.933550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:43.933634 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:43.933920 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:43.933923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:43.933942 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:43.934182 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.934186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.935042 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.935045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:43.935863 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:43.935870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.936574 (+ 704us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.936589 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:57:43.936974 (+ 385us) client_negotiation.cc:240] Negotiation successful
1210 14:57:43.937034 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20251210 14:57:43.937297 7606 raft_consensus.cc:2468] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be79ac2043204dadb558b3ae13a380ae in term 0.
I20251210 14:57:43.937749 7334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e199afd7232542e6b74a8bee9e7687fc" candidate_uuid: "be79ac2043204dadb558b3ae13a380ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e1cdbdab75416da175debd4dfebcc5" is_pre_election: true
I20251210 14:57:43.937799 7406 leader_election.cc:304] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be79ac2043204dadb558b3ae13a380ae, d7b0e6fea63245dda5d20db8f5bec3e6; no voters:
I20251210 14:57:43.937971 7334 raft_consensus.cc:2468] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be79ac2043204dadb558b3ae13a380ae in term 0.
I20251210 14:57:43.938042 7675 raft_consensus.cc:2804] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:43.938115 7675 raft_consensus.cc:493] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:43.938169 7675 raft_consensus.cc:3060] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:43.939123 7675 raft_consensus.cc:515] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.939433 7675 leader_election.cc:290] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [CANDIDATE]: Term 1 election: Requested vote from peers d7b0e6fea63245dda5d20db8f5bec3e6 (127.6.115.67:43701), 03e1cdbdab75416da175debd4dfebcc5 (127.6.115.65:40609)
I20251210 14:57:43.939785 7606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e199afd7232542e6b74a8bee9e7687fc" candidate_uuid: "be79ac2043204dadb558b3ae13a380ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6"
I20251210 14:57:43.939821 7334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e199afd7232542e6b74a8bee9e7687fc" candidate_uuid: "be79ac2043204dadb558b3ae13a380ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e1cdbdab75416da175debd4dfebcc5"
I20251210 14:57:43.939919 7334 raft_consensus.cc:3060] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:43.939927 7606 raft_consensus.cc:3060] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:43.941136 7606 raft_consensus.cc:2468] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be79ac2043204dadb558b3ae13a380ae in term 1.
I20251210 14:57:43.941136 7334 raft_consensus.cc:2468] T e199afd7232542e6b74a8bee9e7687fc P 03e1cdbdab75416da175debd4dfebcc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be79ac2043204dadb558b3ae13a380ae in term 1.
I20251210 14:57:43.941473 7406 leader_election.cc:304] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be79ac2043204dadb558b3ae13a380ae, d7b0e6fea63245dda5d20db8f5bec3e6; no voters:
I20251210 14:57:43.941654 7675 raft_consensus.cc:2804] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:43.941835 7675 raft_consensus.cc:697] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [term 1 LEADER]: Becoming Leader. State: Replica: be79ac2043204dadb558b3ae13a380ae, State: Running, Role: LEADER
I20251210 14:57:43.942098 7675 consensus_queue.cc:237] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } }
I20251210 14:57:43.944921 7187 catalog_manager.cc:5654] T e199afd7232542e6b74a8bee9e7687fc P be79ac2043204dadb558b3ae13a380ae reported cstate change: term changed from 0 to 1, leader changed from <none> to be79ac2043204dadb558b3ae13a380ae (127.6.115.66). New cstate: current_term: 1 leader_uuid: "be79ac2043204dadb558b3ae13a380ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b0e6fea63245dda5d20db8f5bec3e6" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be79ac2043204dadb558b3ae13a380ae" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 35425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e1cdbdab75416da175debd4dfebcc5" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 40609 } health_report { overall_health: UNKNOWN } } }
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:43.974617 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.966732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.6.115.65:40609)
1210 14:57:43.966899 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.966902 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.966982 (+ 80us) server_negotiation.cc:408] Connection header received
1210 14:57:43.967084 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.967086 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.967128 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.967193 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.967945 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.968390 (+ 445us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.969077 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.969301 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.971378 (+ 2077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.971401 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.971412 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.971436 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.973234 (+ 1798us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.973637 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.973642 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.973644 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.973702 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.973885 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.973887 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.973888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.974045 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.974184 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.974295 (+ 111us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.974416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 10 14:57:43 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:43.997514 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:43.990587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.6.115.65:40609)
1210 14:57:43.990733 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:43.990737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:43.990750 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:57:43.990871 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:43.990874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:43.990915 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:43.990975 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:43.991692 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.992143 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:43.992696 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:43.992816 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:43.994699 (+ 1883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:43.994714 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:43.994716 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:43.994741 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:43.996124 (+ 1383us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.996435 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.996438 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.996439 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.996472 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:43.996798 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:43.996801 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:43.996802 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:43.996924 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:43.997019 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:43.997203 (+ 184us) server_negotiation.cc:300] Negotiation successful
1210 14:57:43.997284 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Dec 10 14:57:44 dist-test-slave-nb51 krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 10 14:57:44 dist-test-slave-nb51 krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:44.019743 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:44.012195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.6.115.65:40609)
1210 14:57:44.012304 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:44.012307 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:44.012411 (+ 104us) server_negotiation.cc:408] Connection header received
1210 14:57:44.012654 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:44.012657 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:44.012729 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:44.012828 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:44.013654 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:44.014130 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:44.014709 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:44.014850 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:44.016782 (+ 1932us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:44.016799 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:44.016802 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:44.016828 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:44.018387 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:44.018763 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:44.018766 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:44.018768 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:44.018815 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:44.019007 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:44.019010 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:44.019012 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:44.019129 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:44.019236 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:44.019458 (+ 222us) server_negotiation.cc:300] Negotiation successful
1210 14:57:44.019592 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20251210 14:57:44.021024 7294 tablet_service.cc:3012] Rejecting scan request for tablet e199afd7232542e6b74a8bee9e7687fc: Uninitialized: clean time has not yet been initialized
I20251210 14:57:44.022591 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7251
W20251210 14:57:44.026634 7653 tablet.cc:2378] T e199afd7232542e6b74a8bee9e7687fc P d7b0e6fea63245dda5d20db8f5bec3e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:44.028713 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7387
I20251210 14:57:44.034250 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7523
I20251210 14:57:44.039563 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7155
2025-12-10T14:57:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3212 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:44 dist-test-slave-nb51 krb5kdc[7693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:44 dist-test-slave-nb51 krb5kdc[7693](info): set up 2 sockets
Dec 10 14:57:44 dist-test-slave-nb51 krb5kdc[7693](info): commencing operation
krb5kdc: starting...
W20251210 14:57:46.049539 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20251210 14:57:46.184664 6605 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.135s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:57:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:46.288690 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:42709
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:46.389176 7709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:46.389402 7709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:46.389490 7709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:46.392483 7709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:57:46.392539 7709 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251210 14:57:46.392585 7709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:46.392632 7709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:46.392678 7709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:57:46.392715 7709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:57:46.396693 7709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:42709
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:46.397701 7709 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:46.398496 7709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:46.403715 7714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:46.403784 7715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:46.403729 7717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:46.403986 7709 server_base.cc:1047] running on GCE node
I20251210 14:57:46.404431 7709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:46.405287 7709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:46.406437 7709 hybrid_clock.cc:648] HybridClock initialized: now 1765378666406427 us; error 38 us; skew 500 ppm
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:46.408859 7709 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:57:46.409811 7709 webserver.cc:492] Webserver started at http://127.6.115.126:43435/ using document root <none> and password file <none>
I20251210 14:57:46.410301 7709 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:46.410351 7709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:46.410538 7709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:46.412123 7709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "f24f84606a844d2d8c320b45798682ee"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "60e366e1ba113794dfb145fb3656faa8"
server_key_iv: "ebc2e162b6352045e32c7b8cc59c198f"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.412606 7709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "f24f84606a844d2d8c320b45798682ee"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "60e366e1ba113794dfb145fb3656faa8"
server_key_iv: "ebc2e162b6352045e32c7b8cc59c198f"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.415791 7709 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:46.417830 7724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:46.418751 7709 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:46.418854 7709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "f24f84606a844d2d8c320b45798682ee"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "60e366e1ba113794dfb145fb3656faa8"
server_key_iv: "ebc2e162b6352045e32c7b8cc59c198f"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.418937 7709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:46.438205 7709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:46.440816 7709 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:46.440984 7709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:46.448362 7709 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:42709
I20251210 14:57:46.448583 7786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:42709 every 8 connection(s)
I20251210 14:57:46.449308 7709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:57:46.451871 7787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:46.453413 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7709
I20251210 14:57:46.453516 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:57:46.453743 6605 external_mini_cluster.cc:1468] Setting key 4ac94ccb903b1dbef59b6fd11c7cd082
I20251210 14:57:46.457266 7787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Bootstrap starting.
I20251210 14:57:46.459649 7787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:46.460371 7787 log.cc:826] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Log is configured to *not* fsync() on all Append() calls
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:46.462302 7787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: No bootstrap required, opened a new log
I20251210 14:57:46.465628 7787 raft_consensus.cc:359] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } }
I20251210 14:57:46.465798 7787 raft_consensus.cc:385] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:46.465839 7787 raft_consensus.cc:740] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24f84606a844d2d8c320b45798682ee, State: Initialized, Role: FOLLOWER
I20251210 14:57:46.465930 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.454982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.6.115.126:42709)
1210 14:57:46.455379 (+ 397us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:46.455389 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:46.455418 (+ 29us) server_negotiation.cc:408] Connection header received
1210 14:57:46.455918 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:46.455935 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:46.456263 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:46.456557 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:46.457469 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.458600 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.459230 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.459501 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.461728 (+ 2227us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:46.461744 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:46.461755 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:46.461784 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:46.463525 (+ 1741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.463984 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.463988 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.463992 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.464053 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.464313 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.464315 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.464316 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.464605 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:46.464786 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:46.465048 (+ 262us) server_negotiation.cc:300] Negotiation successful
1210 14:57:46.465293 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20251210 14:57:46.466265 7787 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } }
I20251210 14:57:46.466427 7787 raft_consensus.cc:399] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:57:46.466511 7787 raft_consensus.cc:493] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:57:46.466622 7787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:46.467646 7787 raft_consensus.cc:515] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } }
I20251210 14:57:46.467996 7787 leader_election.cc:304] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24f84606a844d2d8c320b45798682ee; no voters:
I20251210 14:57:46.468256 7787 leader_election.cc:290] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:57:46.468346 7792 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:46.468554 7792 raft_consensus.cc:697] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [term 1 LEADER]: Becoming Leader. State: Replica: f24f84606a844d2d8c320b45798682ee, State: Running, Role: LEADER
I20251210 14:57:46.468879 7792 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } }
I20251210 14:57:46.469274 7787 sys_catalog.cc:565] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:57:46.470376 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24f84606a844d2d8c320b45798682ee. Latest consensus state: current_term: 1 leader_uuid: "f24f84606a844d2d8c320b45798682ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } } }
I20251210 14:57:46.470484 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:46.470734 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f24f84606a844d2d8c320b45798682ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24f84606a844d2d8c320b45798682ee" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 42709 } } }
I20251210 14:57:46.470808 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:46.470834 7801 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:57:46.473652 7801 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:57:46.478202 7801 catalog_manager.cc:1357] Generated new cluster ID: 540c22973d884aff8115973e793f2e69
I20251210 14:57:46.478284 7801 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:57:46.492246 7801 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:57:46.493475 7801 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:57:46.508642 7801 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Generated new TSK 0
I20251210 14:57:46.509496 7801 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:46.554200 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:42709
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:46.655961 7815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:46.656148 7815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:46.656203 7815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:46.659412 7815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:46.659482 7815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:46.659533 7815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:46.659621 7815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:57:46.663727 7815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:46.664700 7815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:46.665551 7815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:46.671873 7821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:46.671868 7820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:46.671963 7815 server_base.cc:1047] running on GCE node
W20251210 14:57:46.671868 7823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:46.672417 7815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:46.672930 7815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:46.674084 7815 hybrid_clock.cc:648] HybridClock initialized: now 1765378666674071 us; error 33 us; skew 500 ppm
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:46.676584 7815 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:57:46.677608 7815 webserver.cc:492] Webserver started at http://127.6.115.65:42229/ using document root <none> and password file <none>
I20251210 14:57:46.678159 7815 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:46.678211 7815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:46.678387 7815 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:46.680033 7815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "fd24e624c7e14bb48867199c6b7394d0"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "83ded36c6dd7dd052da91e8ad99cb44e"
server_key_iv: "8a1366f59c1b05524b6107b5c1aa254e"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.680485 7815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "fd24e624c7e14bb48867199c6b7394d0"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "83ded36c6dd7dd052da91e8ad99cb44e"
server_key_iv: "8a1366f59c1b05524b6107b5c1aa254e"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.683702 7815 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:46.685879 7830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:46.686828 7815 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:57:46.686930 7815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "fd24e624c7e14bb48867199c6b7394d0"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "83ded36c6dd7dd052da91e8ad99cb44e"
server_key_iv: "8a1366f59c1b05524b6107b5c1aa254e"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.687016 7815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:46.696223 7815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:46.699048 7815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:46.699219 7815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:46.700606 7815 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:46.700656 7815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:46.700721 7815 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:46.700757 7815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:46.712090 7815 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:42833
I20251210 14:57:46.712136 7945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:42833 every 8 connection(s)
I20251210 14:57:46.713186 7815 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:57:46.713478 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.701198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:42557 (local address 127.6.115.126:42709)
1210 14:57:46.701351 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:46.701386 (+ 35us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:46.702172 (+ 786us) server_negotiation.cc:408] Connection header received
1210 14:57:46.703245 (+ 1073us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:46.703249 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:46.703315 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:46.703408 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:46.705171 (+ 1763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.706064 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.706981 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.707220 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.709621 (+ 2401us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:46.709644 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:46.709649 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:46.709692 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:46.711400 (+ 1708us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.711985 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.711989 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.711990 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.712047 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.712378 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.712381 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.712383 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.712550 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:46.712642 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:46.713131 (+ 489us) server_negotiation.cc:300] Negotiation successful
1210 14:57:46.713268 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251210 14:57:46.714114 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.701503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.65:42557)
1210 14:57:46.702000 (+ 497us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:46.702051 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:46.703012 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:46.703597 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:46.703610 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:46.704181 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:46.704973 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.704991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.706271 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.706276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:46.706840 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.706849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.707074 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.708018 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:46.708041 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:46.709467 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:46.711515 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.711521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.711531 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.711857 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.712161 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.712164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.712166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.712277 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.712655 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:46.712663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:46.712899 (+ 236us) client_negotiation.cc:769] Sending connection context
1210 14:57:46.713083 (+ 184us) client_negotiation.cc:240] Negotiation successful
1210 14:57:46.713354 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20251210 14:57:46.715291 7946 heartbeater.cc:344] Connected to a master server at 127.6.115.126:42709
I20251210 14:57:46.715438 7946 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:46.716025 7946 heartbeater.cc:507] Master 127.6.115.126:42709 requested a full tablet report, sending...
I20251210 14:57:46.717689 7740 ts_manager.cc:194] Registered new tserver with Master: fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833)
I20251210 14:57:46.718835 7740 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:42557
I20251210 14:57:46.719082 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7815
I20251210 14:57:46.719172 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:57:46.719463 6605 external_mini_cluster.cc:1468] Setting key a9f4f94647fdf72f078334a0f3b69e64
I20251210 14:57:46.724176 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.717605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:40249 (local address 127.6.115.126:42709)
1210 14:57:46.717739 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:46.717744 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:46.717876 (+ 132us) server_negotiation.cc:408] Connection header received
1210 14:57:46.718002 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:46.718006 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:46.718160 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:46.718259 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:46.718784 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.719252 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.719992 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.720161 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.721126 (+ 965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:46.721145 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:46.721151 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:46.721180 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:46.722806 (+ 1626us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.723197 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.723203 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.723205 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.723245 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.723484 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.723487 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.723489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.723619 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:46.723707 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:46.723941 (+ 234us) server_negotiation.cc:300] Negotiation successful
1210 14:57:46.724054 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251210 14:57:46.724184 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.717519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.65:40249)
1210 14:57:46.717794 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:46.717807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:46.717911 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:46.718290 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:46.718293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:46.718470 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:46.718672 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.718678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.719373 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.719376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:46.719878 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.719887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.720000 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.720654 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:46.720668 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:46.720998 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:46.722902 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.722908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.722912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.723088 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.723352 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.723355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.723357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.723407 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.723706 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:46.723709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:46.723795 (+ 86us) client_negotiation.cc:769] Sending connection context
1210 14:57:46.723886 (+ 91us) client_negotiation.cc:240] Negotiation successful
1210 14:57:46.724011 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:46.762714 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:42709
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:46.864779 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:46.864975 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:46.865031 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:46.868292 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:46.868351 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:46.868400 7957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:46.868466 7957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:57:46.872601 7957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:46.873585 7957 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:46.874347 7957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:46.880625 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:46.880657 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:46.880702 7957 server_base.cc:1047] running on GCE node
W20251210 14:57:46.880661 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:46.881271 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:46.881832 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:46.882993 7957 hybrid_clock.cc:648] HybridClock initialized: now 1765378666882978 us; error 39 us; skew 500 ppm
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:46.885593 7957 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:57:46.886574 7957 webserver.cc:492] Webserver started at http://127.6.115.66:42823/ using document root <none> and password file <none>
I20251210 14:57:46.887108 7957 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:46.887163 7957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:46.887377 7957 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:46.889067 7957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "bfcc6c02652840fea7f25725824af71a"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "39ee10f1d73cd45cb7136f6ce1803e95"
server_key_iv: "e1e937c96c1e3c8f91f3fef9d9a960a7"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.889603 7957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "bfcc6c02652840fea7f25725824af71a"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "39ee10f1d73cd45cb7136f6ce1803e95"
server_key_iv: "e1e937c96c1e3c8f91f3fef9d9a960a7"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.892943 7957 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20251210 14:57:46.895157 7972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:46.896195 7957 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:46.896332 7957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "bfcc6c02652840fea7f25725824af71a"
format_stamp: "Formatted at 2025-12-10 14:57:46 on dist-test-slave-nb51"
server_key: "39ee10f1d73cd45cb7136f6ce1803e95"
server_key_iv: "e1e937c96c1e3c8f91f3fef9d9a960a7"
server_key_version: "encryptionkey@0"
I20251210 14:57:46.896425 7957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:46.909126 7957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:46.911968 7957 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:46.912140 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:46.913563 7957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:46.913614 7957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:46.913671 7957 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:46.913703 7957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:46 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:46.924712 7957 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:40769
I20251210 14:57:46.924737 8087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:40769 every 8 connection(s)
I20251210 14:57:46.925812 7957 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:57:46.925858 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.914270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:43971 (local address 127.6.115.126:42709)
1210 14:57:46.914389 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:46.914392 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:46.915203 (+ 811us) server_negotiation.cc:408] Connection header received
1210 14:57:46.916059 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:46.916063 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:46.916126 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:46.916198 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:46.917763 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.918422 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.919351 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.919563 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.922082 (+ 2519us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:46.922099 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:46.922102 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:46.922126 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:46.923976 (+ 1850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.924474 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.924477 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.924479 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.924528 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.924842 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.924846 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.924848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.924991 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:46.925069 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:46.925641 (+ 572us) server_negotiation.cc:300] Negotiation successful
1210 14:57:46.925743 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:57:46.926651 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.914586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.66:43971)
1210 14:57:46.915063 (+ 477us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:46.915096 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:46.915854 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:46.916429 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:46.916440 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:46.916892 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:46.917579 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.917594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.918547 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.918554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:46.919219 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.919234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.919491 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.920165 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:46.920199 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:46.921942 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:46.924089 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.924095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.924105 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.924354 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.924633 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.924637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.924639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.924735 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.925093 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:46.925099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:46.925344 (+ 245us) client_negotiation.cc:769] Sending connection context
1210 14:57:46.925609 (+ 265us) client_negotiation.cc:240] Negotiation successful
1210 14:57:46.925872 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":144,"threads_started":1}
I20251210 14:57:46.927596 8088 heartbeater.cc:344] Connected to a master server at 127.6.115.126:42709
I20251210 14:57:46.927791 8088 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:46.928048 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 7957
I20251210 14:57:46.928169 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:57:46.928311 8088 heartbeater.cc:507] Master 127.6.115.126:42709 requested a full tablet report, sending...
I20251210 14:57:46.928484 6605 external_mini_cluster.cc:1468] Setting key 13c43adbfd16fe769d394546cbaa14bf
I20251210 14:57:46.929275 7741 ts_manager.cc:194] Registered new tserver with Master: bfcc6c02652840fea7f25725824af71a (127.6.115.66:40769)
I20251210 14:57:46.929946 7741 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:43971
I20251210 14:57:46.937628 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.931217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:32985 (local address 127.6.115.126:42709)
1210 14:57:46.931363 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:46.931371 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:46.931386 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:57:46.931438 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:46.931444 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:46.931494 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:46.931582 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:46.932168 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.932670 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.933568 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.933771 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.934711 (+ 940us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:46.934725 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:46.934727 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:46.934748 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:46.936121 (+ 1373us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.936588 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.936591 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.936592 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.936634 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:46.936877 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:46.936879 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:46.936880 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:46.937017 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:46.937090 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:46.937416 (+ 326us) server_negotiation.cc:300] Negotiation successful
1210 14:57:46.937524 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251210 14:57:46.937711 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:46.929850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.66:32985)
1210 14:57:46.930077 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:46.930091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:46.930165 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:46.931640 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:46.931643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:46.931814 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:46.932062 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.932071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.932775 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:46.932779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:46.933417 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:46.933430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:46.933559 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:46.934095 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:46.934116 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:46.934575 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:46.936237 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.936241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.936243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.936457 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.936727 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:46.936732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:46.936734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:46.936793 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:46.937106 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:46.937109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:46.937215 (+ 106us) client_negotiation.cc:769] Sending connection context
1210 14:57:46.937327 (+ 112us) client_negotiation.cc:240] Negotiation successful
1210 14:57:46.937486 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:46.972144 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:42709
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:47.072507 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:47.072686 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:47.072739 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:47.075781 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:47.075835 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:47.075886 8099 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:47.075929 8099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:57:47.079958 8099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:47.080888 8099 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:47.081640 8099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:47.087620 8104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:47.087651 8105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:47.087641 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:47.087862 8099 server_base.cc:1047] running on GCE node
I20251210 14:57:47.088181 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:47.088673 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:47.089816 8099 hybrid_clock.cc:648] HybridClock initialized: now 1765378667089793 us; error 38 us; skew 500 ppm
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378667, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:47.092322 8099 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:57:47.093309 8099 webserver.cc:492] Webserver started at http://127.6.115.67:44115/ using document root <none> and password file <none>
I20251210 14:57:47.093883 8099 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:47.093935 8099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:47.094113 8099 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:47.095679 8099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "ab17081e23b74f35be4f8a511684ed97"
format_stamp: "Formatted at 2025-12-10 14:57:47 on dist-test-slave-nb51"
server_key: "0f21b65e1bedb0f38ea7c480e1ae4335"
server_key_iv: "89496a97510b1a1b3996beef51ff6c87"
server_key_version: "encryptionkey@0"
I20251210 14:57:47.096140 8099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "ab17081e23b74f35be4f8a511684ed97"
format_stamp: "Formatted at 2025-12-10 14:57:47 on dist-test-slave-nb51"
server_key: "0f21b65e1bedb0f38ea7c480e1ae4335"
server_key_iv: "89496a97510b1a1b3996beef51ff6c87"
server_key_version: "encryptionkey@0"
I20251210 14:57:47.099334 8099 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:47.101266 8114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:47.102296 8099 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:47.102397 8099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "ab17081e23b74f35be4f8a511684ed97"
format_stamp: "Formatted at 2025-12-10 14:57:47 on dist-test-slave-nb51"
server_key: "0f21b65e1bedb0f38ea7c480e1ae4335"
server_key_iv: "89496a97510b1a1b3996beef51ff6c87"
server_key_version: "encryptionkey@0"
I20251210 14:57:47.102488 8099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:47.128321 8099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:47.131036 8099 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:47.131217 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:47.132607 8099 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:47.132658 8099 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:47.132716 8099 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:47.132743 8099 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378667, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:47.143393 8099 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:38899
I20251210 14:57:47.143448 8229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:38899 every 8 connection(s)
I20251210 14:57:47.144011 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.133145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:50381 (local address 127.6.115.126:42709)
1210 14:57:47.133301 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.133306 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.133997 (+ 691us) server_negotiation.cc:408] Connection header received
1210 14:57:47.134768 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.134771 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.134828 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.134899 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.136157 (+ 1258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.136735 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.137706 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.137877 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.140098 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.140116 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.140119 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.140142 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.141941 (+ 1799us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.142507 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.142510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.142511 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.142559 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.142910 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.142913 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.142914 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.143093 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.143188 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.143758 (+ 570us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.143868 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:57:47.144415 8099 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:57:47.144548 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.133456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.67:50381)
1210 14:57:47.133865 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.133901 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.134591 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.135049 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.135056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.135524 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:47.136011 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.136022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.136861 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.136865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.137580 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.137590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.137816 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.138528 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:47.138560 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:47.139955 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:47.142047 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.142055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.142068 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.142409 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.142659 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.142662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.142663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.142782 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.143237 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:47.143244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:47.143518 (+ 274us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.143698 (+ 180us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.143883 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20251210 14:57:47.146016 8230 heartbeater.cc:344] Connected to a master server at 127.6.115.126:42709
I20251210 14:57:47.146217 8230 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:47.146775 8230 heartbeater.cc:507] Master 127.6.115.126:42709 requested a full tablet report, sending...
I20251210 14:57:47.147149 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 8099
I20251210 14:57:47.147249 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:57:47.147516 6605 external_mini_cluster.cc:1468] Setting key 250b9c7431c79ad9a48deeaacb84691f
I20251210 14:57:47.148159 7741 ts_manager.cc:194] Registered new tserver with Master: ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899)
I20251210 14:57:47.148654 7741 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:50381
I20251210 14:57:47.150050 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:57:47.154716 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.147922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:45651 (local address 127.6.115.126:42709)
1210 14:57:47.148029 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.148035 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.148268 (+ 233us) server_negotiation.cc:408] Connection header received
1210 14:57:47.148371 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.148374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.148410 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.148473 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.148944 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.149578 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.150318 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.150495 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.151443 (+ 948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.151456 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.151458 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.151478 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.153101 (+ 1623us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.153682 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.153685 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.153686 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.153726 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.154001 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.154004 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.154005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.154133 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.154217 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.154540 (+ 323us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.154621 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:57:47.154857 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.147876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.6.115.67:45651)
1210 14:57:47.148194 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.148205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.148289 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.148492 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.148494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.148646 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:47.148810 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.148815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.149690 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.149694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.150189 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.150198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.150302 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.150907 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:47.150928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:47.151319 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:47.153222 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.153229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.153234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.153527 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.153824 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.153830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.153833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.153903 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.154240 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:47.154250 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:47.154355 (+ 105us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.154483 (+ 128us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.154618 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":196,"threads_started":1}
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:47.173609 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.166772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.6.115.126:42709)
1210 14:57:47.166875 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.166879 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.166890 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:57:47.166919 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.166922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.166955 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.167025 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.167788 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.168283 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.168837 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.168944 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.170970 (+ 2026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.170984 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.170986 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.171006 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.172473 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.172809 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.172811 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.172812 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.172850 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.173001 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.173003 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.173004 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.173143 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.173230 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.173426 (+ 196us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.173505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251210 14:57:47.175886 7741 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:47.178259 7741 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:47.191288 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.186414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:42833 (local address 127.0.0.1:41672)
1210 14:57:47.186793 (+ 379us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.186831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.186963 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.187744 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.187750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.187784 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.188172 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.188183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.189715 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.189718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.190780 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.190789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.190897 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.190946 (+ 49us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.191061 (+ 115us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.191143 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"spinlock_wait_cycles":6656,"thread_start_us":161,"threads_started":1}
I20251210 14:57:47.191314 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.186159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:38899 (local address 127.0.0.1:54328)
1210 14:57:47.186793 (+ 634us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.186831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.187135 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.187744 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.187750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.187786 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.188269 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.188278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.190021 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.190026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.190745 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.190754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.190879 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.190930 (+ 51us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.191023 (+ 93us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.191164 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"spinlock_wait_cycles":7424,"thread_start_us":59,"threads_started":1}
I20251210 14:57:47.191875 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.186338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.6.115.67:38899)
1210 14:57:47.187402 (+ 1064us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.187408 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.187425 (+ 17us) server_negotiation.cc:408] Connection header received
1210 14:57:47.187486 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.187490 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.187601 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.188008 (+ 407us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.188382 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.189880 (+ 1498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.190885 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.191476 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.191559 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.191629 (+ 70us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.191708 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":99,"threads_started":1}
I20251210 14:57:47.192058 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.187796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.6.115.66:40769)
1210 14:57:47.188194 (+ 398us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.188201 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.188410 (+ 209us) server_negotiation.cc:408] Connection header received
1210 14:57:47.188493 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.188496 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.188611 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.188731 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.189067 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.189965 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.191219 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.191682 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.191794 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.191855 (+ 61us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.191929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
I20251210 14:57:47.192291 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.187615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40769 (local address 127.0.0.1:37308)
1210 14:57:47.188277 (+ 662us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.188291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.188384 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.188729 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.188732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.188748 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.188951 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.188958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.190113 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.190116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.190910 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.190919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.191009 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.191021 (+ 12us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.191072 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.191143 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":101,"threads_started":1}
I20251210 14:57:47.193130 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.186799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.6.115.65:42833)
1210 14:57:47.187260 (+ 461us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.187267 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.187289 (+ 22us) server_negotiation.cc:408] Connection header received
1210 14:57:47.187384 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.187389 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.187567 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.187961 (+ 394us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.188297 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.189524 (+ 1227us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.191845 (+ 2321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.192828 (+ 983us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.192896 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.192943 (+ 47us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.193000 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":85,"threads_started":1}
I20251210 14:57:47.194924 8022 tablet_service.cc:1505] Processing CreateTablet for tablet c5a64fae50b748af91a361fc63f0e01c (DEFAULT_TABLE table=test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.194941 8164 tablet_service.cc:1505] Processing CreateTablet for tablet c5a64fae50b748af91a361fc63f0e01c (DEFAULT_TABLE table=test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.195771 8022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a64fae50b748af91a361fc63f0e01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.195871 8164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a64fae50b748af91a361fc63f0e01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.195611 7880 tablet_service.cc:1505] Processing CreateTablet for tablet c5a64fae50b748af91a361fc63f0e01c (DEFAULT_TABLE table=test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.196426 7880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a64fae50b748af91a361fc63f0e01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.200896 8253 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Bootstrap starting.
I20251210 14:57:47.202459 8255 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Bootstrap starting.
I20251210 14:57:47.203086 8253 tablet_bootstrap.cc:654] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.203299 8254 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Bootstrap starting.
I20251210 14:57:47.204173 8253 log.cc:826] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:47.204653 8255 tablet_bootstrap.cc:654] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.205464 8254 tablet_bootstrap.cc:654] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.205749 8253 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: No bootstrap required, opened a new log
I20251210 14:57:47.205910 8253 ts_tablet_manager.cc:1403] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:57:47.205966 8255 log.cc:826] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:47.206635 8254 log.cc:826] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:47.207751 8255 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: No bootstrap required, opened a new log
I20251210 14:57:47.207966 8255 ts_tablet_manager.cc:1403] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20251210 14:57:47.208540 8253 raft_consensus.cc:359] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.208714 8253 raft_consensus.cc:385] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.208753 8253 raft_consensus.cc:740] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcc6c02652840fea7f25725824af71a, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.209120 8253 consensus_queue.cc:260] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.209700 8088 heartbeater.cc:499] Master 127.6.115.126:42709 was elected leader, sending a full tablet report...
I20251210 14:57:47.210815 8254 tablet_bootstrap.cc:492] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: No bootstrap required, opened a new log
I20251210 14:57:47.210645 8255 raft_consensus.cc:359] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.210907 8255 raft_consensus.cc:385] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.210958 8253 ts_tablet_manager.cc:1434] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251210 14:57:47.211052 8254 ts_tablet_manager.cc:1403] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20251210 14:57:47.211051 8255 raft_consensus.cc:740] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd24e624c7e14bb48867199c6b7394d0, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.211472 8255 consensus_queue.cc:260] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.212148 7946 heartbeater.cc:499] Master 127.6.115.126:42709 was elected leader, sending a full tablet report...
I20251210 14:57:47.212378 8255 ts_tablet_manager.cc:1434] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251210 14:57:47.213822 8254 raft_consensus.cc:359] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.214030 8254 raft_consensus.cc:385] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.214105 8254 raft_consensus.cc:740] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab17081e23b74f35be4f8a511684ed97, State: Initialized, Role: FOLLOWER
W20251210 14:57:47.214478 7947 tablet.cc:2378] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:47.214534 8254 consensus_queue.cc:260] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.215219 8254 ts_tablet_manager.cc:1434] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251210 14:57:47.215507 8230 heartbeater.cc:499] Master 127.6.115.126:42709 was elected leader, sending a full tablet report...
I20251210 14:57:47.261381 8259 raft_consensus.cc:493] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:47.261579 8259 raft_consensus.cc:515] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.262760 8259 leader_election.cc:290] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899), fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833)
I20251210 14:57:47.265422 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.262931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:38899 (local address 127.6.115.66:40935)
1210 14:57:47.263072 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.263087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.263189 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.263498 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.263501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.263531 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.263731 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.263736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.264465 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.264471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.265032 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.265039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.265142 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.265164 (+ 22us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.265215 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.265271 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"spinlock_wait_cycles":3456}
I20251210 14:57:47.265663 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.263000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:42833 (local address 127.6.115.66:34329)
1210 14:57:47.263410 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.263422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.263551 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.263788 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.263791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.263806 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.264049 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.264055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.264720 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.264726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.265272 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.265288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.265413 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.265429 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.265481 (+ 52us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.265529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"spinlock_wait_cycles":5888,"thread_start_us":125,"threads_started":1}
I20251210 14:57:47.265925 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.263148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40935 (local address 127.6.115.67:38899)
1210 14:57:47.263310 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.263313 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.263325 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:57:47.263366 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.263368 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.263405 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.263469 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.263854 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.264361 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.265152 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.265667 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.265703 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.265755 (+ 52us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.265815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251210 14:57:47.266227 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.263157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:34329 (local address 127.6.115.65:42833)
1210 14:57:47.263307 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.263312 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.263500 (+ 188us) server_negotiation.cc:408] Connection header received
1210 14:57:47.263655 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.263657 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.263700 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.263769 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.264161 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.264615 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.265412 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.265898 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.265963 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.266018 (+ 55us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.266101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251210 14:57:47.266486 8184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5a64fae50b748af91a361fc63f0e01c" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab17081e23b74f35be4f8a511684ed97" is_pre_election: true
I20251210 14:57:47.266745 8184 raft_consensus.cc:2468] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 0.
I20251210 14:57:47.266757 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5a64fae50b748af91a361fc63f0e01c" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0" is_pre_election: true
I20251210 14:57:47.266956 7900 raft_consensus.cc:2468] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 0.
I20251210 14:57:47.267203 7973 leader_election.cc:304] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab17081e23b74f35be4f8a511684ed97, bfcc6c02652840fea7f25725824af71a; no voters:
I20251210 14:57:47.267457 8259 raft_consensus.cc:2804] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:47.267527 8259 raft_consensus.cc:493] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:47.267609 8259 raft_consensus.cc:3060] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.268529 8259 raft_consensus.cc:515] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.268839 8259 leader_election.cc:290] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 election: Requested vote from peers ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899), fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833)
I20251210 14:57:47.269189 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5a64fae50b748af91a361fc63f0e01c" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0"
I20251210 14:57:47.269189 8184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5a64fae50b748af91a361fc63f0e01c" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab17081e23b74f35be4f8a511684ed97"
I20251210 14:57:47.269341 7900 raft_consensus.cc:3060] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.269341 8184 raft_consensus.cc:3060] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.270641 8184 raft_consensus.cc:2468] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 1.
I20251210 14:57:47.270641 7900 raft_consensus.cc:2468] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 1.
I20251210 14:57:47.270983 7975 leader_election.cc:304] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfcc6c02652840fea7f25725824af71a, fd24e624c7e14bb48867199c6b7394d0; no voters:
I20251210 14:57:47.271183 8259 raft_consensus.cc:2804] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:47.271415 8259 raft_consensus.cc:697] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 1 LEADER]: Becoming Leader. State: Replica: bfcc6c02652840fea7f25725824af71a, State: Running, Role: LEADER
I20251210 14:57:47.271737 8259 consensus_queue.cc:237] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } }
I20251210 14:57:47.274844 7740 catalog_manager.cc:5654] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a reported cstate change: term changed from 0 to 1, leader changed from <none> to bfcc6c02652840fea7f25725824af71a (127.6.115.66). New cstate: current_term: 1 leader_uuid: "bfcc6c02652840fea7f25725824af71a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } health_report { overall_health: HEALTHY } } }
I20251210 14:57:47.290107 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.287590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.6.115.66:40769)
1210 14:57:47.287691 (+ 101us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.287694 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.287706 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:57:47.287871 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.287874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.287919 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.287991 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:47.288278 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.288725 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.289274 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.289441 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.289486 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.289840 (+ 354us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.289916 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.289983 (+ 67us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.290024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251210 14:57:47.294840 8184 raft_consensus.cc:1275] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Refusing update from remote peer bfcc6c02652840fea7f25725824af71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.294917 7900 raft_consensus.cc:1275] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Refusing update from remote peer bfcc6c02652840fea7f25725824af71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.295405 8259 consensus_queue.cc:1048] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.295552 8263 consensus_queue.cc:1048] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.302102 8266 mvcc.cc:204] Tried to move back new op lower bound from 7230991021234573312 to 7230991021148237824. Current Snapshot: MvccSnapshot[applied={T|T < 7230991021234573312}]
I20251210 14:57:47.302822 8267 mvcc.cc:204] Tried to move back new op lower bound from 7230991021234573312 to 7230991021148237824. Current Snapshot: MvccSnapshot[applied={T|T < 7230991021234573312}]
I20251210 14:57:47.310842 7741 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:47.311025 7741 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:47.313196 7741 catalog_manager.cc:5941] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Sending DeleteTablet for 3 replicas of tablet c5a64fae50b748af91a361fc63f0e01c
I20251210 14:57:47.313807 8164 tablet_service.cc:1552] Processing DeleteTablet for tablet c5a64fae50b748af91a361fc63f0e01c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:54328
I20251210 14:57:47.313807 7880 tablet_service.cc:1552] Processing DeleteTablet for tablet c5a64fae50b748af91a361fc63f0e01c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:41672
I20251210 14:57:47.314270 8276 tablet_replica.cc:333] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: stopping tablet replica
I20251210 14:57:47.314241 8022 tablet_service.cc:1552] Processing DeleteTablet for tablet c5a64fae50b748af91a361fc63f0e01c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:37308
I20251210 14:57:47.314512 8276 raft_consensus.cc:2243] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:47.314618 8278 tablet_replica.cc:333] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: stopping tablet replica
I20251210 14:57:47.314826 8276 raft_consensus.cc:2272] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.314888 8278 raft_consensus.cc:2243] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:57:47.315261 8278 raft_consensus.cc:2272] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.315414 7741 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:47.315968 7741 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:47.316118 8276 ts_tablet_manager.cc:1916] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.316581 8278 ts_tablet_manager.cc:1916] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.318836 8276 ts_tablet_manager.cc:1929] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:57:47.318985 8276 log.cc:1199] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/c5a64fae50b748af91a361fc63f0e01c
I20251210 14:57:47.319077 8278 ts_tablet_manager.cc:1929] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:57:47.319185 8278 log.cc:1199] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/c5a64fae50b748af91a361fc63f0e01c
I20251210 14:57:47.319476 8276 ts_tablet_manager.cc:1950] T c5a64fae50b748af91a361fc63f0e01c P fd24e624c7e14bb48867199c6b7394d0: Deleting consensus metadata
I20251210 14:57:47.319482 8278 ts_tablet_manager.cc:1950] T c5a64fae50b748af91a361fc63f0e01c P bfcc6c02652840fea7f25725824af71a: Deleting consensus metadata
I20251210 14:57:47.320658 7727 catalog_manager.cc:4985] TS fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833): tablet c5a64fae50b748af91a361fc63f0e01c (table test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]) successfully deleted
I20251210 14:57:47.320909 7880 tablet_service.cc:1505] Processing CreateTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf (DEFAULT_TABLE table=test-table [id=8c8e13c15f3b4d479e14bd79107789ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.321197 7880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685b8bbe4f04390820f6b5cfc326bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.321218 7726 catalog_manager.cc:4985] TS bfcc6c02652840fea7f25725824af71a (127.6.115.66:40769): tablet c5a64fae50b748af91a361fc63f0e01c (table test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]) successfully deleted
I20251210 14:57:47.321681 8277 tablet_replica.cc:333] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: stopping tablet replica
I20251210 14:57:47.321674 8022 tablet_service.cc:1505] Processing CreateTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf (DEFAULT_TABLE table=test-table [id=8c8e13c15f3b4d479e14bd79107789ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.321866 8022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685b8bbe4f04390820f6b5cfc326bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.322117 8277 raft_consensus.cc:2243] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:47.322443 8277 raft_consensus.cc:2272] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.323165 8255 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Bootstrap starting.
I20251210 14:57:47.323328 8164 tablet_service.cc:1505] Processing CreateTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf (DEFAULT_TABLE table=test-table [id=8c8e13c15f3b4d479e14bd79107789ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:47.323546 8164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685b8bbe4f04390820f6b5cfc326bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.323930 8253 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Bootstrap starting.
I20251210 14:57:47.324052 8255 tablet_bootstrap.cc:654] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.324875 8253 tablet_bootstrap.cc:654] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.325783 8254 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Bootstrap starting.
I20251210 14:57:47.326133 8253 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: No bootstrap required, opened a new log
I20251210 14:57:47.326205 8253 ts_tablet_manager.cc:1403] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251210 14:57:47.326658 8255 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: No bootstrap required, opened a new log
I20251210 14:57:47.326731 8255 ts_tablet_manager.cc:1403] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251210 14:57:47.326679 8253 raft_consensus.cc:359] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.326773 8254 tablet_bootstrap.cc:654] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.326798 8253 raft_consensus.cc:385] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.326830 8253 raft_consensus.cc:740] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcc6c02652840fea7f25725824af71a, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.326962 8253 consensus_queue.cc:260] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.327221 8253 ts_tablet_manager.cc:1434] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251210 14:57:47.327190 8255 raft_consensus.cc:359] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.327311 8255 raft_consensus.cc:385] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.327343 8255 raft_consensus.cc:740] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd24e624c7e14bb48867199c6b7394d0, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.327450 8255 consensus_queue.cc:260] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.327662 8255 ts_tablet_manager.cc:1434] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251210 14:57:47.327939 8277 ts_tablet_manager.cc:1916] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.331360 8254 tablet_bootstrap.cc:492] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: No bootstrap required, opened a new log
I20251210 14:57:47.331447 8254 ts_tablet_manager.cc:1403] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251210 14:57:47.331609 8277 ts_tablet_manager.cc:1929] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:57:47.331686 8277 log.cc:1199] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/c5a64fae50b748af91a361fc63f0e01c
I20251210 14:57:47.331854 8254 raft_consensus.cc:359] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.331956 8254 raft_consensus.cc:385] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.331970 8277 ts_tablet_manager.cc:1950] T c5a64fae50b748af91a361fc63f0e01c P ab17081e23b74f35be4f8a511684ed97: Deleting consensus metadata
I20251210 14:57:47.331993 8254 raft_consensus.cc:740] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab17081e23b74f35be4f8a511684ed97, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.332113 8254 consensus_queue.cc:260] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.332360 8254 ts_tablet_manager.cc:1434] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:47.333025 7725 catalog_manager.cc:4985] TS ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899): tablet c5a64fae50b748af91a361fc63f0e01c (table test-table [id=bdb56ea821854d88bc9fc12e9959b0fe]) successfully deleted
I20251210 14:57:47.375654 8263 raft_consensus.cc:493] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:47.375802 8263 raft_consensus.cc:515] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.376312 8263 leader_election.cc:290] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833), ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899)
I20251210 14:57:47.376675 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1685b8bbe4f04390820f6b5cfc326bbf" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0" is_pre_election: true
I20251210 14:57:47.376740 8184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1685b8bbe4f04390820f6b5cfc326bbf" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab17081e23b74f35be4f8a511684ed97" is_pre_election: true
I20251210 14:57:47.376811 7900 raft_consensus.cc:2468] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 0.
I20251210 14:57:47.376852 8184 raft_consensus.cc:2468] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 0.
I20251210 14:57:47.377130 7975 leader_election.cc:304] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfcc6c02652840fea7f25725824af71a, fd24e624c7e14bb48867199c6b7394d0; no voters:
I20251210 14:57:47.377302 8263 raft_consensus.cc:2804] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:47.377382 8263 raft_consensus.cc:493] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:47.377439 8263 raft_consensus.cc:3060] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.378355 8263 raft_consensus.cc:515] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.378672 8263 leader_election.cc:290] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 election: Requested vote from peers fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833), ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899)
I20251210 14:57:47.379015 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1685b8bbe4f04390820f6b5cfc326bbf" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0"
I20251210 14:57:47.379015 8184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1685b8bbe4f04390820f6b5cfc326bbf" candidate_uuid: "bfcc6c02652840fea7f25725824af71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab17081e23b74f35be4f8a511684ed97"
I20251210 14:57:47.379112 7900 raft_consensus.cc:3060] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.379134 8184 raft_consensus.cc:3060] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.379918 7900 raft_consensus.cc:2468] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 1.
I20251210 14:57:47.380168 8184 raft_consensus.cc:2468] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfcc6c02652840fea7f25725824af71a in term 1.
I20251210 14:57:47.380193 7975 leader_election.cc:304] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfcc6c02652840fea7f25725824af71a, fd24e624c7e14bb48867199c6b7394d0; no voters:
I20251210 14:57:47.380373 8263 raft_consensus.cc:2804] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:47.380456 8263 raft_consensus.cc:697] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 1 LEADER]: Becoming Leader. State: Replica: bfcc6c02652840fea7f25725824af71a, State: Running, Role: LEADER
I20251210 14:57:47.380591 8263 consensus_queue.cc:237] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.382143 7741 catalog_manager.cc:5654] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a reported cstate change: term changed from 0 to 1, leader changed from <none> to bfcc6c02652840fea7f25725824af71a (127.6.115.66). New cstate: current_term: 1 leader_uuid: "bfcc6c02652840fea7f25725824af71a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } health_report { overall_health: UNKNOWN } } }
W20251210 14:57:47.395273 8231 tablet.cc:2378] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Can't schedule compaction. Clean time has not been advanced past its initial value.
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378666, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:47.419196 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.411116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:42709 (local address 127.0.0.1:35484)
1210 14:57:47.411386 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.411398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.411515 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.411748 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.411751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.412221 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:47.412405 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.412412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.413172 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.413176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.413677 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.413687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.413804 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.414509 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:47.414524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:47.415968 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:47.417787 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.417794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.417800 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.418036 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.418265 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:47.418269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:47.418273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:47.418318 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:47.418643 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:47.418653 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:47.418749 (+ 96us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.418853 (+ 104us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.418994 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
I20251210 14:57:47.419195 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.411173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.6.115.126:42709)
1210 14:57:47.411304 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.411309 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.411470 (+ 161us) server_negotiation.cc:408] Connection header received
1210 14:57:47.411608 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.411612 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.411653 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.411728 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.412532 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.413064 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.413792 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.413994 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.416094 (+ 2100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.416108 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.416110 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.416150 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.417678 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.418132 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.418138 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.418140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.418179 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.418399 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.418402 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.418405 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.418539 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.418632 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.418918 (+ 286us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.419045 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
I20251210 14:57:47.423352 7741 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:35484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251210 14:57:47.423991 7741 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251210 14:57:47.426841 8089 tablet.cc:2378] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:47.429394 7880 tablet_service.cc:1505] Processing CreateTablet for tablet 802bba9d5c75450aaacaf08d9b81826f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f55d78987f10458ca48c293f2949cd30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:47.429780 7880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802bba9d5c75450aaacaf08d9b81826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.429740 8164 tablet_service.cc:1505] Processing CreateTablet for tablet 802bba9d5c75450aaacaf08d9b81826f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f55d78987f10458ca48c293f2949cd30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:47.430065 8164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802bba9d5c75450aaacaf08d9b81826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.432143 8254 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: Bootstrap starting.
I20251210 14:57:47.432441 8022 tablet_service.cc:1505] Processing CreateTablet for tablet 802bba9d5c75450aaacaf08d9b81826f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f55d78987f10458ca48c293f2949cd30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:47.432765 8022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802bba9d5c75450aaacaf08d9b81826f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:47.433171 8254 tablet_bootstrap.cc:654] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.433904 8255 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: Bootstrap starting.
I20251210 14:57:47.434543 8254 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: No bootstrap required, opened a new log
I20251210 14:57:47.434621 8254 ts_tablet_manager.cc:1403] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:57:47.435098 8254 raft_consensus.cc:359] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.435187 8255 tablet_bootstrap.cc:654] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.435222 8254 raft_consensus.cc:385] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.435256 8254 raft_consensus.cc:740] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab17081e23b74f35be4f8a511684ed97, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.435357 8254 consensus_queue.cc:260] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.435432 8253 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: Bootstrap starting.
I20251210 14:57:47.435709 8261 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.435822 8261 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.436457 8253 tablet_bootstrap.cc:654] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:47.436913 8254 ts_tablet_manager.cc:1434] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251210 14:57:47.437548 8255 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: No bootstrap required, opened a new log
I20251210 14:57:47.437636 8255 ts_tablet_manager.cc:1403] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251210 14:57:47.438174 8255 raft_consensus.cc:359] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.438282 8255 raft_consensus.cc:385] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.438328 8255 raft_consensus.cc:740] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd24e624c7e14bb48867199c6b7394d0, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.438472 8255 consensus_queue.cc:260] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.438715 8260 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.438860 8260 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.438736 8255 ts_tablet_manager.cc:1434] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:47.439745 8253 tablet_bootstrap.cc:492] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: No bootstrap required, opened a new log
I20251210 14:57:47.439834 8253 ts_tablet_manager.cc:1403] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251210 14:57:47.440332 8253 raft_consensus.cc:359] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.440457 8253 raft_consensus.cc:385] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:47.440495 8253 raft_consensus.cc:740] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcc6c02652840fea7f25725824af71a, State: Initialized, Role: FOLLOWER
I20251210 14:57:47.440615 8253 consensus_queue.cc:260] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.440970 8263 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.441064 8263 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.441255 8253 ts_tablet_manager.cc:1434] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251210 14:57:47.652575 8261 raft_consensus.cc:493] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:47.652706 8261 raft_consensus.cc:515] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.653704 8261 leader_election.cc:290] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833), bfcc6c02652840fea7f25725824af71a (127.6.115.66:40769)
I20251210 14:57:47.657547 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.653932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40769 (local address 127.6.115.67:47613)
1210 14:57:47.654378 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.654391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.654512 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.654862 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.654864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.654878 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.655106 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.655114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.656051 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.656055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.657034 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.657041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.657183 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.657209 (+ 26us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.657270 (+ 61us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.657336 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":163,"thread_start_us":68,"threads_started":1}
I20251210 14:57:47.657843 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.653932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:42833 (local address 127.6.115.67:49767)
1210 14:57:47.654283 (+ 351us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.654303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.654404 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.654687 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.654690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.654709 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:47.655001 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.655008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.656017 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.656023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.657013 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.657027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.657184 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.657209 (+ 25us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.657270 (+ 61us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.657528 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":75,"threads_started":1}
I20251210 14:57:47.657945 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.654045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:49767 (local address 127.6.115.65:42833)
1210 14:57:47.654254 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.654259 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.654377 (+ 118us) server_negotiation.cc:408] Connection header received
1210 14:57:47.654503 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.654507 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.654567 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.654643 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.655119 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.655884 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.657166 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.657664 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.657702 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.657769 (+ 67us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.657822 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20251210 14:57:47.657945 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.654092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:47613 (local address 127.6.115.66:40769)
1210 14:57:47.654253 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.654258 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.654495 (+ 237us) server_negotiation.cc:408] Connection header received
1210 14:57:47.654681 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.654685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.654737 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.654829 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:47.655235 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.655921 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.657166 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.657664 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.657702 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.657768 (+ 66us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.657822 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251210 14:57:47.658469 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802bba9d5c75450aaacaf08d9b81826f" candidate_uuid: "ab17081e23b74f35be4f8a511684ed97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0" is_pre_election: true
I20251210 14:57:47.658505 8042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802bba9d5c75450aaacaf08d9b81826f" candidate_uuid: "ab17081e23b74f35be4f8a511684ed97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfcc6c02652840fea7f25725824af71a" is_pre_election: true
I20251210 14:57:47.658619 7900 raft_consensus.cc:2468] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab17081e23b74f35be4f8a511684ed97 in term 0.
I20251210 14:57:47.658684 8042 raft_consensus.cc:2468] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab17081e23b74f35be4f8a511684ed97 in term 0.
I20251210 14:57:47.659021 8117 leader_election.cc:304] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab17081e23b74f35be4f8a511684ed97, fd24e624c7e14bb48867199c6b7394d0; no voters:
I20251210 14:57:47.659268 8261 raft_consensus.cc:2804] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:47.659341 8261 raft_consensus.cc:493] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:47.659384 8261 raft_consensus.cc:3060] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.660331 8261 raft_consensus.cc:515] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.660650 8261 leader_election.cc:290] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [CANDIDATE]: Term 1 election: Requested vote from peers fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833), bfcc6c02652840fea7f25725824af71a (127.6.115.66:40769)
I20251210 14:57:47.660975 7900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802bba9d5c75450aaacaf08d9b81826f" candidate_uuid: "ab17081e23b74f35be4f8a511684ed97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd24e624c7e14bb48867199c6b7394d0"
I20251210 14:57:47.661044 8042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802bba9d5c75450aaacaf08d9b81826f" candidate_uuid: "ab17081e23b74f35be4f8a511684ed97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfcc6c02652840fea7f25725824af71a"
I20251210 14:57:47.661120 7900 raft_consensus.cc:3060] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.661151 8042 raft_consensus.cc:3060] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:47.661962 7900 raft_consensus.cc:2468] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab17081e23b74f35be4f8a511684ed97 in term 1.
I20251210 14:57:47.662041 8042 raft_consensus.cc:2468] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab17081e23b74f35be4f8a511684ed97 in term 1.
I20251210 14:57:47.662256 8117 leader_election.cc:304] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab17081e23b74f35be4f8a511684ed97, fd24e624c7e14bb48867199c6b7394d0; no voters:
I20251210 14:57:47.662420 8261 raft_consensus.cc:2804] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:47.662582 8261 raft_consensus.cc:697] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [term 1 LEADER]: Becoming Leader. State: Replica: ab17081e23b74f35be4f8a511684ed97, State: Running, Role: LEADER
I20251210 14:57:47.663002 8261 consensus_queue.cc:237] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } }
I20251210 14:57:47.664160 8294 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: TxnStatusTablet state changed. Reason: New leader ab17081e23b74f35be4f8a511684ed97. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.664269 8294 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:47.664556 8296 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251210 14:57:47.664703 8296 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251210 14:57:47.665465 7741 catalog_manager.cc:5654] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab17081e23b74f35be4f8a511684ed97 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } health_report { overall_health: HEALTHY } } }
I20251210 14:57:47.678228 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.674802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.6.115.67:38899)
1210 14:57:47.674910 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.674914 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.675045 (+ 131us) server_negotiation.cc:408] Connection header received
1210 14:57:47.675272 (+ 227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.675275 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.675327 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.675407 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:47.675804 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.676281 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.677153 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.677298 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.677361 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.677858 (+ 497us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.677962 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.678076 (+ 114us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.678132 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251210 14:57:47.678292 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.674731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:38899 (local address 127.0.0.1:54340)
1210 14:57:47.674970 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.674986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.675119 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.675432 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.675436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.675444 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:47.675689 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.675696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.676382 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.676386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.677021 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.677031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.677156 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.677211 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.677969 (+ 758us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.677985 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.678065 (+ 80us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.678163 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20251210 14:57:47.680969 7900 raft_consensus.cc:1275] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Refusing update from remote peer ab17081e23b74f35be4f8a511684ed97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.681180 8042 raft_consensus.cc:1275] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Refusing update from remote peer ab17081e23b74f35be4f8a511684ed97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.681416 8294 consensus_queue.cc:1048] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.681701 8261 consensus_queue.cc:1048] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.682969 8260 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: TxnStatusTablet state changed. Reason: New leader ab17081e23b74f35be4f8a511684ed97. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.683069 8260 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.683185 8263 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: TxnStatusTablet state changed. Reason: New leader ab17081e23b74f35be4f8a511684ed97. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.683279 8263 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.686051 8261 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.686148 8261 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:47.686537 8260 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.686638 8260 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P fd24e624c7e14bb48867199c6b7394d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.698488 8294 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.698603 8294 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P ab17081e23b74f35be4f8a511684ed97: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:47.699783 8263 tablet_replica.cc:442] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab17081e23b74f35be4f8a511684ed97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 } } peers { permanent_uuid: "bfcc6c02652840fea7f25725824af71a" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40769 } } peers { permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 } } }
I20251210 14:57:47.699899 8263 tablet_replica.cc:445] T 802bba9d5c75450aaacaf08d9b81826f P bfcc6c02652840fea7f25725824af71a: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:47.702219 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.696422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:38899 (local address 127.6.115.66:50321)
1210 14:57:47.696760 (+ 338us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.696772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.696866 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.697089 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.697091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.697097 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:47.697300 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.697304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.698299 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.698303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.701308 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.701317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.701592 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.701639 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.702005 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.702009 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.702056 (+ 47us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.702121 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20251210 14:57:47.702239 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.696503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:50321 (local address 127.6.115.67:38899)
1210 14:57:47.696627 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.696631 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.696836 (+ 205us) server_negotiation.cc:408] Connection header received
1210 14:57:47.696947 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.696950 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.697005 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.697093 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:47.697408 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.698192 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.701589 (+ 3397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.701783 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.701828 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.701919 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.701995 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.702065 (+ 70us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.702122 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:57:47.708398 8184 raft_consensus.cc:1275] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Refusing update from remote peer bfcc6c02652840fea7f25725824af71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.708405 7900 raft_consensus.cc:1275] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Refusing update from remote peer bfcc6c02652840fea7f25725824af71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:47.708753 8263 consensus_queue.cc:1048] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd24e624c7e14bb48867199c6b7394d0" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.708880 8259 consensus_queue.cc:1048] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab17081e23b74f35be4f8a511684ed97" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:47.712810 8268 mvcc.cc:204] Tried to move back new op lower bound from 7230991022930759680 to 7230991021593157632. Current Snapshot: MvccSnapshot[applied={T|T < 7230991022930759680}]
I20251210 14:57:47.729822 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.726157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:35875 (local address 127.6.115.66:40769)
1210 14:57:47.726259 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.726264 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.726577 (+ 313us) server_negotiation.cc:408] Connection header received
1210 14:57:47.726744 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.726747 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.726793 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.726837 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:47.727371 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.728055 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.728956 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.729146 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.729388 (+ 242us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.729469 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.729513 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.729657 (+ 144us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.729705 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251210 14:57:47.730177 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.725825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40769 (local address 127.6.115.67:35875)
1210 14:57:47.726500 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:47.726515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:47.726643 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:47.726953 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:47.726955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:47.726962 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:47.727261 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.727267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.728193 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.728196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:47.728839 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:47.728847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.729211 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.729248 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:47.729570 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:47.729575 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:57:47.729979 (+ 404us) client_negotiation.cc:240] Negotiation successful
1210 14:57:47.730042 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":98,"threads_started":1}
I20251210 14:57:47.753228 7741 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:47.753376 7741 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:47.755214 7741 catalog_manager.cc:5941] T 00000000000000000000000000000000 P f24f84606a844d2d8c320b45798682ee: Sending DeleteTablet for 3 replicas of tablet 1685b8bbe4f04390820f6b5cfc326bbf
I20251210 14:57:47.755733 8022 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:37308
I20251210 14:57:47.755753 7880 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:41672
I20251210 14:57:47.755892 8278 tablet_replica.cc:333] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: stopping tablet replica
I20251210 14:57:47.755909 8276 tablet_replica.cc:333] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: stopping tablet replica
I20251210 14:57:47.756014 8276 raft_consensus.cc:2243] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:47.756023 8278 raft_consensus.cc:2243] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:57:47.756130 8276 raft_consensus.cc:2272] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.756204 8278 raft_consensus.cc:2272] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.756424 8164 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685b8bbe4f04390820f6b5cfc326bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:47 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:54328
I20251210 14:57:47.756695 8277 tablet_replica.cc:333] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: stopping tablet replica
I20251210 14:57:47.756779 8277 raft_consensus.cc:2243] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:47.756870 8277 raft_consensus.cc:2272] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:47.756989 8278 ts_tablet_manager.cc:1916] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.758041 8277 ts_tablet_manager.cc:1916] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.758129 8276 ts_tablet_manager.cc:1916] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:47.759598 8278 ts_tablet_manager.cc:1929] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:47.759657 8278 log.cc:1199] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/1685b8bbe4f04390820f6b5cfc326bbf
I20251210 14:57:47.759917 8278 ts_tablet_manager.cc:1950] T 1685b8bbe4f04390820f6b5cfc326bbf P bfcc6c02652840fea7f25725824af71a: Deleting consensus metadata
I20251210 14:57:47.760394 8277 ts_tablet_manager.cc:1929] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:47.760394 8276 ts_tablet_manager.cc:1929] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:47.760454 8277 log.cc:1199] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/1685b8bbe4f04390820f6b5cfc326bbf
I20251210 14:57:47.760454 8276 log.cc:1199] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/1685b8bbe4f04390820f6b5cfc326bbf
I20251210 14:57:47.760694 8277 ts_tablet_manager.cc:1950] T 1685b8bbe4f04390820f6b5cfc326bbf P ab17081e23b74f35be4f8a511684ed97: Deleting consensus metadata
I20251210 14:57:47.760694 8276 ts_tablet_manager.cc:1950] T 1685b8bbe4f04390820f6b5cfc326bbf P fd24e624c7e14bb48867199c6b7394d0: Deleting consensus metadata
I20251210 14:57:47.761226 7727 catalog_manager.cc:4985] TS fd24e624c7e14bb48867199c6b7394d0 (127.6.115.65:42833): tablet 1685b8bbe4f04390820f6b5cfc326bbf (table test-table [id=8c8e13c15f3b4d479e14bd79107789ff]) successfully deleted
I20251210 14:57:47.761310 7725 catalog_manager.cc:4985] TS ab17081e23b74f35be4f8a511684ed97 (127.6.115.67:38899): tablet 1685b8bbe4f04390820f6b5cfc326bbf (table test-table [id=8c8e13c15f3b4d479e14bd79107789ff]) successfully deleted
I20251210 14:57:47.761550 7726 catalog_manager.cc:4985] TS bfcc6c02652840fea7f25725824af71a (127.6.115.66:40769): tablet 1685b8bbe4f04390820f6b5cfc326bbf (table test-table [id=8c8e13c15f3b4d479e14bd79107789ff]) successfully deleted
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[7693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:57:47.771544 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.761820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.6.115.65:42833)
1210 14:57:47.761932 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.761936 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.762091 (+ 155us) server_negotiation.cc:408] Connection header received
1210 14:57:47.762216 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.762225 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.762268 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.762328 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.763091 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.763804 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.764642 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.764966 (+ 324us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.767945 (+ 2979us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.767966 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.767980 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.768005 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.770045 (+ 2040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.770525 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.770529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.770531 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.770586 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.770792 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.770794 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.770796 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.770959 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.771127 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.771281 (+ 154us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.771382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20251210 14:57:47.771940 7940 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41676
I20251210 14:57:47.779664 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:47.773472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.6.115.126:42709)
1210 14:57:47.773604 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:47.773609 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:47.773721 (+ 112us) server_negotiation.cc:408] Connection header received
1210 14:57:47.773838 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:47.773842 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:47.773890 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:47.773958 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:47.774642 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.775112 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:47.775682 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:47.775884 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:47.776862 (+ 978us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:47.776877 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:47.776879 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:47.776899 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:47.778352 (+ 1453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.778710 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.778713 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.778714 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.778753 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:47.778990 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:47.778994 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:47.778995 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:47.779121 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:47.779234 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:47.779416 (+ 182us) server_negotiation.cc:300] Negotiation successful
1210 14:57:47.779518 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20251210 14:57:47.779976 7741 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494
I20251210 14:57:47.780997 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7815
I20251210 14:57:47.787768 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7957
I20251210 14:57:47.796464 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 8099
I20251210 14:57:47.802402 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 7709
2025-12-10T14:57:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3762 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[8316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[8316](info): set up 2 sockets
Dec 10 14:57:47 dist-test-slave-nb51 krb5kdc[8316](info): commencing operation
krb5kdc: starting...
W20251210 14:57:49.769622 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:57:49 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:57:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:49.898350 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45499
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:49.998762 8332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:49.998991 8332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:49.999053 8332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:50.002324 8332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:57:50.002375 8332 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251210 14:57:50.002393 8332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:50.002414 8332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:50.002439 8332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:57:50.002478 8332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:57:50.006659 8332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:45499
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:50.007596 8332 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:50.008387 8332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:50.013696 8338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.013689 8337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.013690 8340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:50.013865 8332 server_base.cc:1047] running on GCE node
I20251210 14:57:50.014416 8332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:50.015261 8332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:50.016424 8332 hybrid_clock.cc:648] HybridClock initialized: now 1765378670016401 us; error 37 us; skew 500 ppm
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:50.018998 8332 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:57:50.019964 8332 webserver.cc:492] Webserver started at http://127.6.115.126:34273/ using document root <none> and password file <none>
I20251210 14:57:50.020452 8332 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:50.020498 8332 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:50.020645 8332 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:50.022367 8332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "5e0de6dc53cd4126ae23a51ea7bf0724"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "dbaf958d2931da475ac27b14add184df"
server_key_iv: "416d4e6571d315855424172263f94e62"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.022809 8332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "5e0de6dc53cd4126ae23a51ea7bf0724"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "dbaf958d2931da475ac27b14add184df"
server_key_iv: "416d4e6571d315855424172263f94e62"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.026038 8332 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251210 14:57:50.028225 8347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.029184 8332 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251210 14:57:50.029297 8332 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "5e0de6dc53cd4126ae23a51ea7bf0724"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "dbaf958d2931da475ac27b14add184df"
server_key_iv: "416d4e6571d315855424172263f94e62"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.029418 8332 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:50.042117 8332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:50.047833 8332 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:50.048010 8332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:50.056471 8332 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:45499
I20251210 14:57:50.056468 8409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:45499 every 8 connection(s)
I20251210 14:57:50.057533 8332 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:57:50.060474 8410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.062988 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 8332
I20251210 14:57:50.063105 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:57:50.063337 6605 external_mini_cluster.cc:1468] Setting key f185bfa7031bf06d70e8513e87fbaef5
I20251210 14:57:50.065853 8410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Bootstrap starting.
I20251210 14:57:50.068333 8410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.069134 8410 log.cc:826] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Log is configured to *not* fsync() on all Append() calls
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.071074 8410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: No bootstrap required, opened a new log
I20251210 14:57:50.074052 8410 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } }
I20251210 14:57:50.074229 8410 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.074293 8410 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0de6dc53cd4126ae23a51ea7bf0724, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.074740 8410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } }
I20251210 14:57:50.074867 8410 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:57:50.074944 8410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:57:50.075078 8410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.075631 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.064517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.6.115.126:45499)
1210 14:57:50.064876 (+ 359us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.064885 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.064918 (+ 33us) server_negotiation.cc:408] Connection header received
1210 14:57:50.065435 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.065453 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.065740 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.066035 (+ 295us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.067037 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.067758 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.068632 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.068841 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.071259 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.071276 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.071285 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.071315 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.072987 (+ 1672us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.073475 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.073479 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.073483 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.073565 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.073876 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.073881 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.073885 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.074220 (+ 335us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.074444 (+ 224us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.074759 (+ 315us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.075061 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20251210 14:57:50.075896 8410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } }
I20251210 14:57:50.076215 8410 leader_election.cc:304] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e0de6dc53cd4126ae23a51ea7bf0724; no voters:
I20251210 14:57:50.076488 8410 leader_election.cc:290] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:57:50.076630 8415 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:50.076807 8415 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [term 1 LEADER]: Becoming Leader. State: Replica: 5e0de6dc53cd4126ae23a51ea7bf0724, State: Running, Role: LEADER
I20251210 14:57:50.077081 8415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } }
I20251210 14:57:50.077683 8410 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:57:50.078262 8417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e0de6dc53cd4126ae23a51ea7bf0724. Latest consensus state: current_term: 1 leader_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } } }
I20251210 14:57:50.078380 8417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:50.078583 8416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0de6dc53cd4126ae23a51ea7bf0724" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45499 } } }
I20251210 14:57:50.078667 8416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724 [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:50.078750 8420 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:57:50.081332 8420 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:57:50.086241 8420 catalog_manager.cc:1357] Generated new cluster ID: 02f21aa7f7ff4e53855bdf6f226cf425
I20251210 14:57:50.086303 8420 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:57:50.094792 8420 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:57:50.095611 8420 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:57:50.105378 8420 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Generated new TSK 0
I20251210 14:57:50.106016 8420 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:50.149806 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45499
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:50.252327 8438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:50.252516 8438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:50.252571 8438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:50.255671 8438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:50.255723 8438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:50.255775 8438 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251210 14:57:50.255796 8438 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:50.255836 8438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:57:50.259763 8438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:50.260655 8438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:50.261416 8438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:50.267364 8446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.267385 8443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.267364 8444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:50.267715 8438 server_base.cc:1047] running on GCE node
I20251210 14:57:50.268159 8438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:50.268716 8438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:50.269865 8438 hybrid_clock.cc:648] HybridClock initialized: now 1765378670269843 us; error 37 us; skew 500 ppm
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:50.272251 8438 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:57:50.273227 8438 webserver.cc:492] Webserver started at http://127.6.115.65:46009/ using document root <none> and password file <none>
I20251210 14:57:50.273763 8438 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:50.273824 8438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:50.273993 8438 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:50.275616 8438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "652a49856e4a43f886ed450c4189bffb"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "85902b5a87164558af030f778b41c876"
server_key_iv: "2a74442b64ededf736836727f47ec155"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.276065 8438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "652a49856e4a43f886ed450c4189bffb"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "85902b5a87164558af030f778b41c876"
server_key_iv: "2a74442b64ededf736836727f47ec155"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.279238 8438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:50.281205 8453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.282127 8438 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:50.282225 8438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "652a49856e4a43f886ed450c4189bffb"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "85902b5a87164558af030f778b41c876"
server_key_iv: "2a74442b64ededf736836727f47ec155"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.282305 8438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:50.301020 8438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:50.303578 8438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:50.303738 8438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:50.305097 8438 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:50.305168 8438 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.305222 8438 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:50.305255 8438 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.315886 8438 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:35533
I20251210 14:57:50.316210 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.305659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:41839 (local address 127.6.115.126:45499)
1210 14:57:50.305802 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.305806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.306574 (+ 768us) server_negotiation.cc:408] Connection header received
1210 14:57:50.307378 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.307382 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.307441 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.307518 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.308843 (+ 1325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.309400 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.310033 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.310237 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.312750 (+ 2513us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.312769 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.312771 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.312795 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.314269 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.314747 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.314751 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.314755 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.314800 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.315056 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.315059 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.315061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.315203 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.315291 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.315922 (+ 631us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.316061 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:57:50.316417 8568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:35533 every 8 connection(s)
I20251210 14:57:50.316601 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.305973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.65:41839)
1210 14:57:50.306441 (+ 468us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.306473 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.307199 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.307685 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.307692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.308156 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.308694 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.308713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.309513 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.309517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.309925 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.309933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.310154 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.310761 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.310797 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.312613 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.314401 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.314406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.314416 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.314652 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.314884 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.314887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.314889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.314967 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.315300 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.315308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.315504 (+ 196us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.315696 (+ 192us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.315946 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20251210 14:57:50.316953 8438 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:57:50.318430 8569 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45499
I20251210 14:57:50.318583 8569 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:50.319033 8569 heartbeater.cc:507] Master 127.6.115.126:45499 requested a full tablet report, sending...
I20251210 14:57:50.320369 8364 ts_manager.cc:194] Registered new tserver with Master: 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533)
I20251210 14:57:50.321512 8364 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:41839
I20251210 14:57:50.324927 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 8438
I20251210 14:57:50.325068 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:57:50.325342 6605 external_mini_cluster.cc:1468] Setting key afba0170ad3c6f728529255da16be25c
I20251210 14:57:50.326077 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.319840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.65:39685)
1210 14:57:50.320104 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.320117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.320225 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.320447 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.320450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.320612 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.320812 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.320818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.321639 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.321643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.322072 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.322084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.322181 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.322786 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.322800 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.323079 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.324669 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.324673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.324675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.324857 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.325153 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.325157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.325159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.325222 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.325641 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.325644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.325731 (+ 87us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.325826 (+ 95us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.325917 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20251210 14:57:50.326308 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.319935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:39685 (local address 127.6.115.126:45499)
1210 14:57:50.320053 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.320058 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.320183 (+ 125us) server_negotiation.cc:408] Connection header received
1210 14:57:50.320322 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.320325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.320362 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.320424 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.320919 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.321545 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.322168 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.322364 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.323216 (+ 852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.323232 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.323236 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.323264 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.324552 (+ 1288us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.324963 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.324966 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.324967 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.325027 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.325330 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.325333 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.325334 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.325542 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.325630 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.326026 (+ 396us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.326127 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:50.372658 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45499
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:50.474562 8580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:50.474817 8580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:50.474911 8580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:50.478134 8580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:50.478190 8580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:50.478276 8580 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251210 14:57:50.478304 8580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:50.478404 8580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:57:50.482425 8580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:50.483392 8580 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:50.484207 8580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:50.490283 8586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.490255 8585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.490273 8588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:50.490481 8580 server_base.cc:1047] running on GCE node
I20251210 14:57:50.490851 8580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:50.491379 8580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:50.492523 8580 hybrid_clock.cc:648] HybridClock initialized: now 1765378670492509 us; error 41 us; skew 500 ppm
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:50.495076 8580 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:57:50.496028 8580 webserver.cc:492] Webserver started at http://127.6.115.66:43351/ using document root <none> and password file <none>
I20251210 14:57:50.496553 8580 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:50.496608 8580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:50.496788 8580 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:50.498498 8580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "dbde94ed1791422b99fc4da5380764c9"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "fc1a55df0432318a9ba7f56a63c802aa"
server_key_iv: "38bf914e923af42554fc483f29b709ec"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.498975 8580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "dbde94ed1791422b99fc4da5380764c9"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "fc1a55df0432318a9ba7f56a63c802aa"
server_key_iv: "38bf914e923af42554fc483f29b709ec"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.502139 8580 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:57:50.504223 8595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.505152 8580 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:50.505252 8580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "dbde94ed1791422b99fc4da5380764c9"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "fc1a55df0432318a9ba7f56a63c802aa"
server_key_iv: "38bf914e923af42554fc483f29b709ec"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.505336 8580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:50.518851 8580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:50.521688 8580 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:50.521893 8580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:50.523228 8580 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:50.523278 8580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.523356 8580 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:50.523387 8580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.534040 8580 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:39573
I20251210 14:57:50.534080 8710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:39573 every 8 connection(s)
I20251210 14:57:50.535051 8580 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:57:50.535416 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.523947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:39895 (local address 127.6.115.126:45499)
1210 14:57:50.524078 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.524082 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.524838 (+ 756us) server_negotiation.cc:408] Connection header received
1210 14:57:50.525759 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.525763 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.525827 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.525903 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.527609 (+ 1706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.528456 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.529236 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.529427 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.531531 (+ 2104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.531548 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.531550 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.531575 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.533399 (+ 1824us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.533968 (+ 569us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.533973 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.533977 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.534054 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.534366 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.534370 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.534372 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.534552 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.534662 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.535107 (+ 445us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.535226 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:57:50.535980 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.524186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.66:39895)
1210 14:57:50.524689 (+ 503us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.524722 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.525548 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.526173 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.526184 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.526682 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.527418 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.527433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.528563 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.528567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.529129 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.529141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.529342 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.529952 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.529982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.531374 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.533515 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.533523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.533536 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.533862 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.534178 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.534184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.534187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.534270 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.534649 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.534654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.534861 (+ 207us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.535021 (+ 160us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.535254 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
I20251210 14:57:50.536970 8711 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45499
I20251210 14:57:50.537124 8711 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:50.537590 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 8580
I20251210 14:57:50.537658 8711 heartbeater.cc:507] Master 127.6.115.126:45499 requested a full tablet report, sending...
I20251210 14:57:50.537734 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:57:50.538026 6605 external_mini_cluster.cc:1468] Setting key d6307ff52e181ba0b18ddf4049e22880
I20251210 14:57:50.538647 8363 ts_manager.cc:194] Registered new tserver with Master: dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573)
I20251210 14:57:50.539364 8363 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:39895
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:57:50.547060 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.539176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:43339 (local address 127.6.115.126:45499)
1210 14:57:50.540975 (+ 1799us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.540980 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.540993 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:57:50.541035 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.541038 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.541077 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.541151 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.541785 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.542270 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.543149 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.543268 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.544256 (+ 988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.544269 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.544272 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.544292 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.545631 (+ 1339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.546072 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.546074 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.546075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.546113 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.546349 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.546351 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.546352 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.546489 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.546560 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.546862 (+ 302us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.546942 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1721}
I20251210 14:57:50.547153 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.539197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.66:43339)
1210 14:57:50.539487 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.539503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.539587 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.541261 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.541264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.541478 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.541672 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.541678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.542400 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.542407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.543019 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.543031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.543152 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.543664 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.543684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.544124 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.545735 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.545742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.545747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.545979 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.546193 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.546196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.546199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.546262 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.546583 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.546590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.546697 (+ 107us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.546819 (+ 122us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.546935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:50.581930 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45499
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:50.685794 8722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:50.686066 8722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:50.686163 8722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:50.689393 8722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:50.689442 8722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:50.689493 8722 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251210 14:57:50.689525 8722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:57:50.689599 8722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:57:50.693610 8722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:50.694554 8722 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:50.695344 8722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:50.701293 8727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.701304 8728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:50.701388 8730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:50.701613 8722 server_base.cc:1047] running on GCE node
I20251210 14:57:50.701962 8722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:50.702472 8722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:50.703627 8722 hybrid_clock.cc:648] HybridClock initialized: now 1765378670703582 us; error 74 us; skew 500 ppm
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:50.706233 8722 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:57:50.707228 8722 webserver.cc:492] Webserver started at http://127.6.115.67:45089/ using document root <none> and password file <none>
I20251210 14:57:50.707738 8722 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:50.707791 8722 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:50.707986 8722 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:50.709690 8722 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "d93fedee92c5499da73abb9cf6254e03"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "8c13eb1885f497624dd59d5aebcd60dd"
server_key_iv: "ed9063bebb80d098b8c120e9606c2a6d"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.710160 8722 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "d93fedee92c5499da73abb9cf6254e03"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "8c13eb1885f497624dd59d5aebcd60dd"
server_key_iv: "ed9063bebb80d098b8c120e9606c2a6d"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.713303 8722 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:57:50.715386 8737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.716418 8722 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251210 14:57:50.716532 8722 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "d93fedee92c5499da73abb9cf6254e03"
format_stamp: "Formatted at 2025-12-10 14:57:50 on dist-test-slave-nb51"
server_key: "8c13eb1885f497624dd59d5aebcd60dd"
server_key_iv: "ed9063bebb80d098b8c120e9606c2a6d"
server_key_version: "encryptionkey@0"
I20251210 14:57:50.716619 8722 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:50.734964 8722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:50.737752 8722 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:50.737924 8722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:50.739357 8722 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:50.739411 8722 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:50.739468 8722 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:50.739503 8722 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.750452 8722 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:39775
I20251210 14:57:50.750491 8852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:39775 every 8 connection(s)
I20251210 14:57:50.751068 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.739886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:33607 (local address 127.6.115.126:45499)
1210 14:57:50.740022 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.740027 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.740768 (+ 741us) server_negotiation.cc:408] Connection header received
1210 14:57:50.741649 (+ 881us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.741653 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.741709 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.741787 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.743220 (+ 1433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.743791 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.744690 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.744974 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.747266 (+ 2292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.747286 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.747289 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.747317 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.748957 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.749572 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.749575 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.749578 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.749641 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.749965 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.749968 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.749970 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.750126 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.750210 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.750797 (+ 587us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.750918 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:57:50.751533 8722 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:57:50.751542 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.740186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.67:33607)
1210 14:57:50.740625 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.740664 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.741446 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.741949 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.741956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.742454 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.743082 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.743095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.743911 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.743914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.744543 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.744554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.744774 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.745448 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.745479 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.747132 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.749093 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.749100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.749114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.749453 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.749747 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.749750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.749752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.749861 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.750261 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.750266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.750511 (+ 245us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.750699 (+ 188us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.750901 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20251210 14:57:50.752802 8853 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45499
I20251210 14:57:50.752954 8853 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:50.753438 8853 heartbeater.cc:507] Master 127.6.115.126:45499 requested a full tablet report, sending...
I20251210 14:57:50.754383 8363 ts_manager.cc:194] Registered new tserver with Master: d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:50.755077 8363 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:33607
I20251210 14:57:50.756912 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 8722
I20251210 14:57:50.757015 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:57:50.757232 6605 external_mini_cluster.cc:1468] Setting key a639c132afdebd4867ffb770c1e74af7
I20251210 14:57:50.759239 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:57:50.760849 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.754558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.6.115.67:33753)
1210 14:57:50.754778 (+ 220us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.754793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.754885 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.755300 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.755306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.755452 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.755703 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.755712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.756393 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.756396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.756943 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.756952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.757063 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.757640 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.757655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.757937 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.759685 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.759690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.759692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.759856 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.760080 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.760085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.760087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.760127 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.760408 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.760411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.760485 (+ 74us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.760572 (+ 87us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.760689 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20251210 14:57:50.760849 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.754870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:33753 (local address 127.6.115.126:45499)
1210 14:57:50.755002 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.755007 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.755037 (+ 30us) server_negotiation.cc:408] Connection header received
1210 14:57:50.755070 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.755073 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.755117 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.755198 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.755844 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.756288 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.757077 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.757292 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.758058 (+ 766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.758074 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.758077 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.758099 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.759585 (+ 1486us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.759957 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.759960 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.759961 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.760004 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.760202 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.760205 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.760207 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.760324 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.760407 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.760627 (+ 220us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.760729 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.782428 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.774738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.6.115.126:45499)
1210 14:57:50.774864 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.774868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.774878 (+ 10us) server_negotiation.cc:408] Connection header received
1210 14:57:50.774909 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.774911 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.774946 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.775041 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.775962 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.776629 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.777470 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.778097 (+ 627us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.779968 (+ 1871us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.779983 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.779984 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.780005 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.781228 (+ 1223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.781632 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.781635 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.781637 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.781681 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.781870 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.781872 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.781873 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.781989 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.782082 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.782238 (+ 156us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.782318 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:57:50.784651 8363 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:50.787055 8363 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:50.799681 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.795424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:39573 (local address 127.0.0.1:53096)
1210 14:57:50.796387 (+ 963us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.796396 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.796479 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.797213 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.797216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.797233 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.797488 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.797495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798605 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.798608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.799180 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.799187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.799293 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.799345 (+ 52us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.799454 (+ 109us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.799537 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":109,"threads_started":1}
I20251210 14:57:50.800426 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.795691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.6.115.66:39573)
1210 14:57:50.796167 (+ 476us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.796174 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.796478 (+ 304us) server_negotiation.cc:408] Connection header received
1210 14:57:50.796905 (+ 427us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.796911 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.797097 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.797220 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.797597 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798456 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.799311 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.799852 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.799969 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.800198 (+ 229us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.800290 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":99,"threads_started":1}
I20251210 14:57:50.800676 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.795666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.6.115.67:39775)
1210 14:57:50.796041 (+ 375us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.796047 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.796272 (+ 225us) server_negotiation.cc:408] Connection header received
1210 14:57:50.796354 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.796359 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.796512 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.796659 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.797079 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798247 (+ 1168us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.799754 (+ 1507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.800286 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.800386 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.800452 (+ 66us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.800531 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20251210 14:57:50.800676 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.794979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.6.115.65:35533)
1210 14:57:50.795257 (+ 278us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.795268 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.795987 (+ 719us) server_negotiation.cc:408] Connection header received
1210 14:57:50.796212 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.796219 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.796409 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.796580 (+ 171us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.797120 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798252 (+ 1132us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.799710 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.800299 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.800386 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.800452 (+ 66us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.800531 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
I20251210 14:57:50.800961 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.794791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35533 (local address 127.0.0.1:45678)
1210 14:57:50.795832 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.795871 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.796030 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.796598 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.796606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.796629 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.796967 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.796977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798395 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.798399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.799322 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.799333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.799478 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.799498 (+ 20us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.799559 (+ 61us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.799621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":69,"threads_started":1}
I20251210 14:57:50.801187 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.795374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.0.0.1:34586)
1210 14:57:50.796183 (+ 809us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.796194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.796263 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.796639 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.796642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.796654 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.796954 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.796962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.798395 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.798399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.799322 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.799334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.799479 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.799498 (+ 19us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.799568 (+ 70us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.799621 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"spinlock_wait_cycles":896,"thread_start_us":114,"threads_started":1}
I20251210 14:57:50.803440 8645 tablet_service.cc:1505] Processing CreateTablet for tablet 897f76daa26c459c98f2391ba82c1fcc (DEFAULT_TABLE table=test-table [id=6475a85936304b2cb78e62da5aca64ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:50.803550 8787 tablet_service.cc:1505] Processing CreateTablet for tablet 897f76daa26c459c98f2391ba82c1fcc (DEFAULT_TABLE table=test-table [id=6475a85936304b2cb78e62da5aca64ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:50.804425 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897f76daa26c459c98f2391ba82c1fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.804425 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897f76daa26c459c98f2391ba82c1fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.806203 8503 tablet_service.cc:1505] Processing CreateTablet for tablet 897f76daa26c459c98f2391ba82c1fcc (DEFAULT_TABLE table=test-table [id=6475a85936304b2cb78e62da5aca64ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:50.806955 8503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897f76daa26c459c98f2391ba82c1fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.810738 8876 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Bootstrap starting.
I20251210 14:57:50.811409 8877 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Bootstrap starting.
I20251210 14:57:50.812815 8878 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Bootstrap starting.
I20251210 14:57:50.813123 8876 tablet_bootstrap.cc:654] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.813149 8877 tablet_bootstrap.cc:654] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.814294 8876 log.cc:826] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:50.814373 8877 log.cc:826] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:50.814939 8878 tablet_bootstrap.cc:654] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.816044 8878 log.cc:826] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:50.816267 8876 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: No bootstrap required, opened a new log
I20251210 14:57:50.816471 8876 ts_tablet_manager.cc:1403] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251210 14:57:50.817840 8878 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: No bootstrap required, opened a new log
I20251210 14:57:50.818025 8878 ts_tablet_manager.cc:1403] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:57:50.819377 8876 raft_consensus.cc:359] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.819605 8876 raft_consensus.cc:385] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.819664 8876 raft_consensus.cc:740] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.820088 8876 consensus_queue.cc:260] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.820797 8711 heartbeater.cc:499] Master 127.6.115.126:45499 was elected leader, sending a full tablet report...
I20251210 14:57:50.820978 8876 ts_tablet_manager.cc:1434] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251210 14:57:50.821400 8878 raft_consensus.cc:359] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.821580 8878 raft_consensus.cc:385] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.821664 8878 raft_consensus.cc:740] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.822053 8878 consensus_queue.cc:260] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.822211 8877 tablet_bootstrap.cc:492] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: No bootstrap required, opened a new log
I20251210 14:57:50.822414 8877 ts_tablet_manager.cc:1403] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20251210 14:57:50.822845 8878 ts_tablet_manager.cc:1434] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:57:50.823211 8569 heartbeater.cc:499] Master 127.6.115.126:45499 was elected leader, sending a full tablet report...
I20251210 14:57:50.824980 8877 raft_consensus.cc:359] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.825135 8877 raft_consensus.cc:385] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.825235 8877 raft_consensus.cc:740] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.825654 8877 consensus_queue.cc:260] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.826193 8853 heartbeater.cc:499] Master 127.6.115.126:45499 was elected leader, sending a full tablet report...
I20251210 14:57:50.826395 8877 ts_tablet_manager.cc:1434] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251210 14:57:50.831115 8883 raft_consensus.cc:493] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:50.831218 8883 raft_consensus.cc:515] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.832008 8883 leader_election.cc:290] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:50.834877 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.832184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.6.115.65:39335)
1210 14:57:50.832386 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.832401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.832496 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.832831 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.832833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.832842 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.833051 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.833057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.833931 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.833934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.834533 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.834541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.834632 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.834648 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.834693 (+ 45us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.834735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20251210 14:57:50.835050 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.832184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:39573 (local address 127.6.115.65:50349)
1210 14:57:50.832675 (+ 491us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.832686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.832744 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.832983 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.832986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.833003 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.833256 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.833264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.834051 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.834055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.834666 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.834675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.834791 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.834809 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.834860 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.834913 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":7,"thread_start_us":73,"threads_started":1}
I20251210 14:57:50.835451 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.832601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:50349 (local address 127.6.115.66:39573)
1210 14:57:50.832723 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.832729 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.832744 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:57:50.832824 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.832827 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.832871 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.832959 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.833399 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.833922 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.834796 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.835263 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.835291 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.835331 (+ 40us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.835367 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:57:50.835529 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.832481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:39335 (local address 127.6.115.67:39775)
1210 14:57:50.832588 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.832592 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.832603 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:57:50.832640 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.832646 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.832691 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.832767 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.833156 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.833822 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.834665 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.835232 (+ 567us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.835276 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.835323 (+ 47us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.835379 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251210 14:57:50.835969 8665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897f76daa26c459c98f2391ba82c1fcc" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9" is_pre_election: true
I20251210 14:57:50.835979 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897f76daa26c459c98f2391ba82c1fcc" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03" is_pre_election: true
I20251210 14:57:50.836195 8798 raft_consensus.cc:2468] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 0.
I20251210 14:57:50.836195 8665 raft_consensus.cc:2468] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 0.
I20251210 14:57:50.836632 8457 leader_election.cc:304] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, d93fedee92c5499da73abb9cf6254e03; no voters:
I20251210 14:57:50.836841 8883 raft_consensus.cc:2804] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:50.836910 8883 raft_consensus.cc:493] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:50.836946 8883 raft_consensus.cc:3060] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.837844 8883 raft_consensus.cc:515] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.838210 8883 leader_election.cc:290] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 election: Requested vote from peers dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:50.838505 8665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897f76daa26c459c98f2391ba82c1fcc" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9"
I20251210 14:57:50.838553 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897f76daa26c459c98f2391ba82c1fcc" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03"
I20251210 14:57:50.838614 8665 raft_consensus.cc:3060] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.838644 8798 raft_consensus.cc:3060] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.839511 8798 raft_consensus.cc:2468] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 1.
I20251210 14:57:50.839561 8665 raft_consensus.cc:2468] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 1.
I20251210 14:57:50.839785 8457 leader_election.cc:304] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, d93fedee92c5499da73abb9cf6254e03; no voters:
I20251210 14:57:50.839984 8883 raft_consensus.cc:2804] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:50.840183 8883 raft_consensus.cc:697] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 1 LEADER]: Becoming Leader. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Running, Role: LEADER
I20251210 14:57:50.840476 8883 consensus_queue.cc:237] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.843555 8364 catalog_manager.cc:5654] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a49856e4a43f886ed450c4189bffb (127.6.115.65). New cstate: current_term: 1 leader_uuid: "652a49856e4a43f886ed450c4189bffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } health_report { overall_health: UNKNOWN } } }
Dec 10 14:57:50 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:50.894177 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.887009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45499 (local address 127.0.0.1:45492)
1210 14:57:50.887284 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.887300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.887408 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.887635 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.887637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.888062 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:50.888246 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.888252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.888899 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.888902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.889257 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.889263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.889352 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.890018 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:50.890030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:50.891452 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:50.892960 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.892967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.892971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.893173 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.893402 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:50.893407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:50.893409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:50.893453 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:50.893728 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:50.893733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:50.893820 (+ 87us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.893907 (+ 87us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.894012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20251210 14:57:50.894197 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.887078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.6.115.126:45499)
1210 14:57:50.887229 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.887233 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.887372 (+ 139us) server_negotiation.cc:408] Connection header received
1210 14:57:50.887480 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.887483 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.887522 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.887599 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:50.888340 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.888810 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.889338 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.889503 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.891573 (+ 2070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:50.891588 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:50.891590 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:50.891612 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:50.892865 (+ 1253us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.893253 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.893255 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.893256 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.893294 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:50.893518 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:50.893520 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:50.893521 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:50.893649 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:50.893721 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.893945 (+ 224us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.894080 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":55}
I20251210 14:57:50.897753 8364 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:45492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251210 14:57:50.898365 8364 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:50.903185 8645 tablet_service.cc:1505] Processing CreateTablet for tablet 0e72c8f38ed140fe8d67b54e32038dda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f473e3e0a14a464891c66c7cdb2a979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:50.903184 8503 tablet_service.cc:1505] Processing CreateTablet for tablet 0e72c8f38ed140fe8d67b54e32038dda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f473e3e0a14a464891c66c7cdb2a979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:50.903407 8787 tablet_service.cc:1505] Processing CreateTablet for tablet 0e72c8f38ed140fe8d67b54e32038dda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f473e3e0a14a464891c66c7cdb2a979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:57:50.903817 8503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e72c8f38ed140fe8d67b54e32038dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.903825 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e72c8f38ed140fe8d67b54e32038dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.903985 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e72c8f38ed140fe8d67b54e32038dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:50.906419 8877 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: Bootstrap starting.
I20251210 14:57:50.906500 8878 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: Bootstrap starting.
I20251210 14:57:50.906569 8876 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: Bootstrap starting.
I20251210 14:57:50.907318 8878 tablet_bootstrap.cc:654] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.907526 8877 tablet_bootstrap.cc:654] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.907552 8876 tablet_bootstrap.cc:654] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:50.908635 8878 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: No bootstrap required, opened a new log
I20251210 14:57:50.908730 8878 ts_tablet_manager.cc:1403] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:50.909076 8876 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: No bootstrap required, opened a new log
I20251210 14:57:50.909160 8876 ts_tablet_manager.cc:1403] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251210 14:57:50.909160 8878 raft_consensus.cc:359] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.909257 8878 raft_consensus.cc:385] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.909288 8878 raft_consensus.cc:740] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.909477 8877 tablet_bootstrap.cc:492] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: No bootstrap required, opened a new log
I20251210 14:57:50.909439 8878 consensus_queue.cc:260] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.909550 8877 ts_tablet_manager.cc:1403] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20251210 14:57:50.909600 8876 raft_consensus.cc:359] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.909727 8876 raft_consensus.cc:385] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.909737 8878 ts_tablet_manager.cc:1434] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251210 14:57:50.909760 8876 raft_consensus.cc:740] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.909718 8883 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.909829 8883 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:50.909876 8876 consensus_queue.cc:260] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.910025 8877 raft_consensus.cc:359] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.910133 8877 raft_consensus.cc:385] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:50.910135 8876 ts_tablet_manager.cc:1434] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:50.910168 8877 raft_consensus.cc:740] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Initialized, Role: FOLLOWER
I20251210 14:57:50.910336 8877 consensus_queue.cc:260] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.910444 8882 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.910558 8882 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:50.910619 8877 ts_tablet_manager.cc:1434] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251210 14:57:50.910837 8884 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.910974 8884 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:50.960738 8884 raft_consensus.cc:493] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:50.960875 8884 raft_consensus.cc:515] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.961884 8884 leader_election.cc:290] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573)
I20251210 14:57:50.965116 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.962092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35533 (local address 127.6.115.67:43459)
1210 14:57:50.962251 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.962268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.962364 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.962669 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.962672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.962694 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.962947 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.962953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.963906 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.963909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.964722 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.964732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.964834 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.964856 (+ 22us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.964905 (+ 49us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.964966 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20251210 14:57:50.965382 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.962363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:39573 (local address 127.6.115.67:44053)
1210 14:57:50.962706 (+ 343us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.962718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.962815 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.963039 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.963043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.963078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:50.963291 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.963296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.964117 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.964120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.964953 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.964960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.965108 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.965123 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.965178 (+ 55us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.965233 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":137,"threads_started":1}
I20251210 14:57:50.965641 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.962187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:43459 (local address 127.6.115.65:35533)
1210 14:57:50.962345 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.962351 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.962366 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:57:50.962478 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.962482 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.962527 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.962597 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.963084 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.963783 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.964869 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.965381 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.965421 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.965470 (+ 49us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.965526 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251210 14:57:50.965791 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.962513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:44053 (local address 127.6.115.66:39573)
1210 14:57:50.962660 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.962665 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.962783 (+ 118us) server_negotiation.cc:408] Connection header received
1210 14:57:50.962898 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.962900 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.962937 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.963014 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:50.963384 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.964027 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.965092 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.965539 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.965570 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.965627 (+ 57us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.965685 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:57:50.966133 8665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e72c8f38ed140fe8d67b54e32038dda" candidate_uuid: "d93fedee92c5499da73abb9cf6254e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9" is_pre_election: true
I20251210 14:57:50.966176 8523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e72c8f38ed140fe8d67b54e32038dda" candidate_uuid: "d93fedee92c5499da73abb9cf6254e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb" is_pre_election: true
I20251210 14:57:50.966290 8665 raft_consensus.cc:2468] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93fedee92c5499da73abb9cf6254e03 in term 0.
I20251210 14:57:50.966354 8523 raft_consensus.cc:2468] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93fedee92c5499da73abb9cf6254e03 in term 0.
I20251210 14:57:50.966660 8739 leader_election.cc:304] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d93fedee92c5499da73abb9cf6254e03, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:50.966907 8884 raft_consensus.cc:2804] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:50.966969 8884 raft_consensus.cc:493] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:50.967025 8884 raft_consensus.cc:3060] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.967865 8884 raft_consensus.cc:515] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.968160 8884 leader_election.cc:290] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [CANDIDATE]: Term 1 election: Requested vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573)
I20251210 14:57:50.968502 8523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e72c8f38ed140fe8d67b54e32038dda" candidate_uuid: "d93fedee92c5499da73abb9cf6254e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb"
I20251210 14:57:50.968588 8665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e72c8f38ed140fe8d67b54e32038dda" candidate_uuid: "d93fedee92c5499da73abb9cf6254e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9"
I20251210 14:57:50.968633 8523 raft_consensus.cc:3060] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.968698 8665 raft_consensus.cc:3060] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:50.969980 8665 raft_consensus.cc:2468] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93fedee92c5499da73abb9cf6254e03 in term 1.
I20251210 14:57:50.970263 8739 leader_election.cc:304] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d93fedee92c5499da73abb9cf6254e03, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:50.970302 8523 raft_consensus.cc:2468] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93fedee92c5499da73abb9cf6254e03 in term 1.
I20251210 14:57:50.970430 8884 raft_consensus.cc:2804] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:50.970695 8884 raft_consensus.cc:697] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [term 1 LEADER]: Becoming Leader. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Running, Role: LEADER
I20251210 14:57:50.970991 8884 consensus_queue.cc:237] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:50.972981 8899 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: TxnStatusTablet state changed. Reason: New leader d93fedee92c5499da73abb9cf6254e03. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.973083 8899 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:50.973464 8901 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251210 14:57:50.973647 8901 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251210 14:57:50.973980 8364 catalog_manager.cc:5654] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d93fedee92c5499da73abb9cf6254e03 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } health_report { overall_health: HEALTHY } } }
I20251210 14:57:50.989346 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.986092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.0.0.1:34598)
1210 14:57:50.986230 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:50.986247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:50.986360 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:50.986683 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:50.986688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:50.986695 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:50.986857 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.986862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.987768 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.987771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:50.988163 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:50.988169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.988252 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.988287 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:50.989111 (+ 824us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:50.989124 (+ 13us) client_negotiation.cc:769] Sending connection context
1210 14:57:50.989181 (+ 57us) client_negotiation.cc:240] Negotiation successful
1210 14:57:50.989244 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20251210 14:57:50.989406 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:50.986173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.6.115.67:39775)
1210 14:57:50.986292 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:50.986298 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:50.986316 (+ 18us) server_negotiation.cc:408] Connection header received
1210 14:57:50.986481 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:50.986484 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:50.986548 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:50.986650 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:50.986951 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.987639 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:50.988260 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:50.988438 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:50.988494 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:50.989007 (+ 513us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:50.989122 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:50.989209 (+ 87us) server_negotiation.cc:300] Negotiation successful
1210 14:57:50.989265 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251210 14:57:50.993974 8523 raft_consensus.cc:1275] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Refusing update from remote peer d93fedee92c5499da73abb9cf6254e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:50.993978 8665 raft_consensus.cc:1275] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Refusing update from remote peer d93fedee92c5499da73abb9cf6254e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:50.994606 8884 consensus_queue.cc:1048] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:50.994807 8899 consensus_queue.cc:1048] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:50.998303 8883 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: TxnStatusTablet state changed. Reason: New leader d93fedee92c5499da73abb9cf6254e03. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.998418 8883 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:50.998608 8882 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: TxnStatusTablet state changed. Reason: New leader d93fedee92c5499da73abb9cf6254e03. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:50.998724 8882 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: This TxnStatusTablet replica's current role is: FOLLOWER
W20251210 14:57:51.002554 8854 tablet.cc:2378] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:51.002637 8899 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:51.002720 8899 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:51.002739 8884 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:51.002841 8884 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P d93fedee92c5499da73abb9cf6254e03: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:57:51.003099 8906 mvcc.cc:204] Tried to move back new op lower bound from 7230991036385308672 to 7230991036300099584. Current Snapshot: MvccSnapshot[applied={T|T < 7230991036385308672}]
I20251210 14:57:51.003412 8904 mvcc.cc:204] Tried to move back new op lower bound from 7230991036385308672 to 7230991036300099584. Current Snapshot: MvccSnapshot[applied={T|T < 7230991036385308672}]
I20251210 14:57:51.004076 8882 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:51.004179 8882 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P dbde94ed1791422b99fc4da5380764c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:51.004961 8883 tablet_replica.cc:442] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d93fedee92c5499da73abb9cf6254e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } } }
I20251210 14:57:51.005061 8883 tablet_replica.cc:445] T 0e72c8f38ed140fe8d67b54e32038dda P 652a49856e4a43f886ed450c4189bffb: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:57:51.012504 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.009174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.6.115.65:35533)
1210 14:57:51.009311 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.009316 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.009329 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:57:51.009419 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.009423 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.009481 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.009577 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.009960 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.010717 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.011344 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.011524 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.011580 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.012077 (+ 497us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.012186 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.012325 (+ 139us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.012379 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251210 14:57:51.023234 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.019871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.6.115.65:50889)
1210 14:57:51.020177 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.020194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.020315 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.020753 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.020755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.020763 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:51.021059 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.021067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.022040 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.022043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.022505 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.022513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.022599 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.022630 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.023013 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.023017 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.023069 (+ 52us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.023123 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20251210 14:57:51.023255 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.020347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:50889 (local address 127.6.115.67:39775)
1210 14:57:51.020486 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.020491 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.020505 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:57:51.020552 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.020555 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.020610 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.020688 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.021162 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.021915 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.022612 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.022806 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.022850 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.022924 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.023000 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.023092 (+ 92us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.023136 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251210 14:57:51.029996 8665 raft_consensus.cc:1275] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Refusing update from remote peer 652a49856e4a43f886ed450c4189bffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:51.030130 8798 raft_consensus.cc:1275] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Refusing update from remote peer 652a49856e4a43f886ed450c4189bffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:51.030504 8887 consensus_queue.cc:1048] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:51.030817 8883 consensus_queue.cc:1048] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:51.036810 8905 mvcc.cc:204] Tried to move back new op lower bound from 7230991036535492608 to 7230991035765264384. Current Snapshot: MvccSnapshot[applied={T|T < 7230991036535492608}]
I20251210 14:57:51.059275 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.055415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35533 (local address 127.6.115.67:50219)
1210 14:57:51.055628 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.055644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.055734 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.056211 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.056214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.056222 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:51.056488 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.056495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.057635 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.057640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.058323 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.058333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.058444 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.058490 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.058941 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.058949 (+ 8us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.059021 (+ 72us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.059098 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20251210 14:57:51.059986 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.055809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:50219 (local address 127.6.115.65:35533)
1210 14:57:51.055930 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.055935 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.055948 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:57:51.055988 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.055992 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.056043 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.056111 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.056613 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.057436 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.058450 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.058694 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.058742 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.058824 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.058898 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.059769 (+ 871us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.059824 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251210 14:57:51.097297 8364 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:51.097483 8364 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:51.100344 8364 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Sending DeleteTablet for 3 replicas of tablet 897f76daa26c459c98f2391ba82c1fcc
I20251210 14:57:51.101284 8503 tablet_service.cc:1552] Processing DeleteTablet for tablet 897f76daa26c459c98f2391ba82c1fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:51 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:45678
I20251210 14:57:51.101534 8645 tablet_service.cc:1552] Processing DeleteTablet for tablet 897f76daa26c459c98f2391ba82c1fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:51 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:53096
I20251210 14:57:51.101778 8787 tablet_service.cc:1552] Processing DeleteTablet for tablet 897f76daa26c459c98f2391ba82c1fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:51 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:34586
I20251210 14:57:51.102717 8364 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:51.103118 8364 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:51.108112 8787 tablet_service.cc:1505] Processing CreateTablet for tablet 73fafae580174dd0bcd911f783b23c98 (DEFAULT_TABLE table=test-table [id=d371400857224357b259b216ddbfec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:51.108382 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fafae580174dd0bcd911f783b23c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:51.109568 8927 tablet_replica.cc:333] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: stopping tablet replica
I20251210 14:57:51.109870 8927 raft_consensus.cc:2243] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:51.110052 8927 raft_consensus.cc:2272] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:51.110594 8877 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Bootstrap starting.
I20251210 14:57:51.111456 8877 tablet_bootstrap.cc:654] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:51.111514 8927 ts_tablet_manager.cc:1916] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:51.112953 8503 tablet_service.cc:1505] Processing CreateTablet for tablet 73fafae580174dd0bcd911f783b23c98 (DEFAULT_TABLE table=test-table [id=d371400857224357b259b216ddbfec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:51.113247 8503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fafae580174dd0bcd911f783b23c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:51.113427 8645 tablet_service.cc:1505] Processing CreateTablet for tablet 73fafae580174dd0bcd911f783b23c98 (DEFAULT_TABLE table=test-table [id=d371400857224357b259b216ddbfec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:51.113664 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fafae580174dd0bcd911f783b23c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:51.115736 8876 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Bootstrap starting.
I20251210 14:57:51.115876 8878 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Bootstrap starting.
I20251210 14:57:51.116322 8925 tablet_replica.cc:333] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: stopping tablet replica
I20251210 14:57:51.116547 8925 raft_consensus.cc:2243] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:57:51.116717 8876 tablet_bootstrap.cc:654] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:51.116864 8878 tablet_bootstrap.cc:654] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:51.116904 8925 raft_consensus.cc:2272] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:51.117175 8926 tablet_replica.cc:333] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: stopping tablet replica
I20251210 14:57:51.117404 8926 raft_consensus.cc:2243] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:51.117733 8926 raft_consensus.cc:2272] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:51.117894 8878 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: No bootstrap required, opened a new log
I20251210 14:57:51.117980 8878 ts_tablet_manager.cc:1403] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251210 14:57:51.118464 8878 raft_consensus.cc:359] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.118595 8878 raft_consensus.cc:385] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:51.118628 8878 raft_consensus.cc:740] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Initialized, Role: FOLLOWER
I20251210 14:57:51.118743 8878 consensus_queue.cc:260] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.118983 8878 ts_tablet_manager.cc:1434] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251210 14:57:51.120606 8927 ts_tablet_manager.cc:1929] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:51.121157 8925 ts_tablet_manager.cc:1916] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:51.122932 8926 ts_tablet_manager.cc:1916] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:51.124491 8925 ts_tablet_manager.cc:1929] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:51.126338 8926 ts_tablet_manager.cc:1929] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:51.126478 8926 log.cc:1199] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/897f76daa26c459c98f2391ba82c1fcc
I20251210 14:57:51.126899 8926 ts_tablet_manager.cc:1950] T 897f76daa26c459c98f2391ba82c1fcc P dbde94ed1791422b99fc4da5380764c9: Deleting consensus metadata
I20251210 14:57:51.128706 8349 catalog_manager.cc:4985] TS dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573): tablet 897f76daa26c459c98f2391ba82c1fcc (table test-table [id=6475a85936304b2cb78e62da5aca64ba]) successfully deleted
I20251210 14:57:51.129472 8925 log.cc:1199] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/897f76daa26c459c98f2391ba82c1fcc
I20251210 14:57:51.125425 8877 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: No bootstrap required, opened a new log
I20251210 14:57:51.129652 8927 log.cc:1199] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/897f76daa26c459c98f2391ba82c1fcc
I20251210 14:57:51.129727 8877 ts_tablet_manager.cc:1403] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20251210 14:57:51.129956 8927 ts_tablet_manager.cc:1950] T 897f76daa26c459c98f2391ba82c1fcc P d93fedee92c5499da73abb9cf6254e03: Deleting consensus metadata
I20251210 14:57:51.130329 8877 raft_consensus.cc:359] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.130471 8877 raft_consensus.cc:385] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:51.130510 8877 raft_consensus.cc:740] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Initialized, Role: FOLLOWER
I20251210 14:57:51.130654 8877 consensus_queue.cc:260] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.130954 8351 catalog_manager.cc:4985] TS d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775): tablet 897f76daa26c459c98f2391ba82c1fcc (table test-table [id=6475a85936304b2cb78e62da5aca64ba]) successfully deleted
I20251210 14:57:51.131003 8877 ts_tablet_manager.cc:1434] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:51.131349 8925 ts_tablet_manager.cc:1950] T 897f76daa26c459c98f2391ba82c1fcc P 652a49856e4a43f886ed450c4189bffb: Deleting consensus metadata
I20251210 14:57:51.132951 8349 catalog_manager.cc:4985] TS 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533): tablet 897f76daa26c459c98f2391ba82c1fcc (table test-table [id=6475a85936304b2cb78e62da5aca64ba]) successfully deleted
I20251210 14:57:51.133826 8876 tablet_bootstrap.cc:492] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: No bootstrap required, opened a new log
I20251210 14:57:51.133927 8876 ts_tablet_manager.cc:1403] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20251210 14:57:51.134397 8876 raft_consensus.cc:359] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.134495 8876 raft_consensus.cc:385] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:51.134527 8876 raft_consensus.cc:740] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Initialized, Role: FOLLOWER
I20251210 14:57:51.134649 8876 consensus_queue.cc:260] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.134898 8876 ts_tablet_manager.cc:1434] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251210 14:57:51.268466 8882 raft_consensus.cc:493] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:51.268607 8882 raft_consensus.cc:515] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.269800 8882 leader_election.cc:290] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:51.274360 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.269972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35533 (local address 127.6.115.66:50509)
1210 14:57:51.270455 (+ 483us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.270472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.270578 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.271145 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.271148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.271163 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:51.271449 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.271460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.272466 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.272470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.273325 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.273333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.274069 (+ 736us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.274086 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.274156 (+ 70us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.274207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":70,"threads_started":1}
I20251210 14:57:51.274844 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.270632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:50509 (local address 127.6.115.65:35533)
1210 14:57:51.270780 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.270785 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.270799 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:57:51.270854 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.270857 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.270908 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.270983 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:51.271640 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.272337 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.273477 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.273949 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.273991 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.274667 (+ 676us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.274729 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251210 14:57:51.275481 8523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73fafae580174dd0bcd911f783b23c98" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb" is_pre_election: true
I20251210 14:57:51.275626 8523 raft_consensus.cc:2468] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 0.
I20251210 14:57:51.275856 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.270754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:41067 (local address 127.6.115.67:39775)
1210 14:57:51.271164 (+ 410us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.271168 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.271182 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:57:51.271228 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.271234 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.271285 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.271390 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:57:51.272341 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.273077 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.274483 (+ 1406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.275026 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.275060 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.275563 (+ 503us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.275630 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20251210 14:57:51.276273 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.270135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.6.115.66:41067)
1210 14:57:51.270958 (+ 823us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.270974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.271044 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.271487 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.271493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.271922 (+ 429us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:57:51.272201 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.272209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.273303 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.273307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.274328 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.274344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.275182 (+ 838us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.275379 (+ 197us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.276047 (+ 668us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.276120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":58,"threads_started":1}
I20251210 14:57:51.276422 8597 leader_election.cc:304] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:51.276728 8882 raft_consensus.cc:2804] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:51.276784 8882 raft_consensus.cc:493] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:51.276827 8882 raft_consensus.cc:3060] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:51.276995 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73fafae580174dd0bcd911f783b23c98" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03" is_pre_election: true
I20251210 14:57:51.277112 8798 raft_consensus.cc:2468] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 0.
I20251210 14:57:51.285038 8882 raft_consensus.cc:515] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:51.285542 8882 leader_election.cc:290] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 election: Requested vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:51.286329 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73fafae580174dd0bcd911f783b23c98" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03"
I20251210 14:57:51.286463 8798 raft_consensus.cc:3060] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:51.287827 8798 raft_consensus.cc:2468] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 1.
I20251210 14:57:51.288244 8599 leader_election.cc:304] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d93fedee92c5499da73abb9cf6254e03, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:51.289515 8882 raft_consensus.cc:2804] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:51.289695 8882 raft_consensus.cc:697] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 1 LEADER]: Becoming Leader. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Running, Role: LEADER
I20251210 14:57:51.289952 8882 consensus_queue.cc:237] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
W20251210 14:57:51.293226 8712 tablet.cc:2378] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:51.294121 8523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73fafae580174dd0bcd911f783b23c98" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb"
I20251210 14:57:51.294247 8523 raft_consensus.cc:3060] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:51.295534 8364 catalog_manager.cc:5654] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbde94ed1791422b99fc4da5380764c9 (127.6.115.66). New cstate: current_term: 1 leader_uuid: "dbde94ed1791422b99fc4da5380764c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } health_report { overall_health: UNKNOWN } } }
I20251210 14:57:51.301844 8523 raft_consensus.cc:2468] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 1.
I20251210 14:57:51.318187 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.314034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.6.115.66:39573)
1210 14:57:51.314144 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.314148 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.314160 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:57:51.314240 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.314243 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.314303 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.314405 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.314998 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.315738 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.316738 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.316926 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.316980 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.317590 (+ 610us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.317693 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.318023 (+ 330us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.318070 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20251210 14:57:51.318926 8570 tablet.cc:2378] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:57:51.334344 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.330327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:43643 (local address 127.6.115.67:39775)
1210 14:57:51.330493 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.330498 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.330808 (+ 310us) server_negotiation.cc:408] Connection header received
1210 14:57:51.330983 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.330987 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.331051 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.331096 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.331688 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.332415 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.333429 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.333630 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.333918 (+ 288us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.333996 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.334037 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.334176 (+ 139us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.334222 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20251210 14:57:51.334915 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.329996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:39775 (local address 127.6.115.66:43643)
1210 14:57:51.330733 (+ 737us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.330748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.330874 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.331214 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.331217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.331225 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:51.331524 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.331555 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.332548 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.332552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.333251 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.333278 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.333720 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.333795 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.334090 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.334095 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.334704 (+ 609us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.334770 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":96,"threads_started":1}
I20251210 14:57:51.341841 8798 raft_consensus.cc:1275] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Refusing update from remote peer dbde94ed1791422b99fc4da5380764c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:51.342165 8522 raft_consensus.cc:1275] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Refusing update from remote peer dbde94ed1791422b99fc4da5380764c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:51.342736 8882 consensus_queue.cc:1048] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:51.345804 8882 consensus_queue.cc:1048] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:51.367936 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.365164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:39573 (local address 127.6.115.67:56119)
1210 14:57:51.365340 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:51.365395 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:51.365485 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:51.365825 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:51.365827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:51.365834 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:57:51.366033 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.366041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.366790 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.366794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:51.367186 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:51.367193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.367298 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.367339 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.367687 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.367695 (+ 8us) client_negotiation.cc:769] Sending connection context
1210 14:57:51.367756 (+ 61us) client_negotiation.cc:240] Negotiation successful
1210 14:57:51.367834 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20251210 14:57:51.368525 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:51.365295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:56119 (local address 127.6.115.66:39573)
1210 14:57:51.365440 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:51.365445 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:51.365458 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:57:51.365604 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:51.365607 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:51.365659 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:51.365726 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:51.366150 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.366687 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:51.367302 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:51.367484 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:51.367522 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:51.367594 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:51.367665 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:51.368358 (+ 693us) server_negotiation.cc:300] Negotiation successful
1210 14:57:51.368411 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20251210 14:57:51.369181 8905 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251210 14:57:52.929474 8364 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:52.929615 8364 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45490:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:52.931896 8364 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Sending DeleteTablet for 3 replicas of tablet 73fafae580174dd0bcd911f783b23c98
I20251210 14:57:52.932420 8503 tablet_service.cc:1552] Processing DeleteTablet for tablet 73fafae580174dd0bcd911f783b23c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:52 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:45678
I20251210 14:57:52.932471 8787 tablet_service.cc:1552] Processing DeleteTablet for tablet 73fafae580174dd0bcd911f783b23c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:52 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:34586
I20251210 14:57:52.932538 8645 tablet_service.cc:1552] Processing DeleteTablet for tablet 73fafae580174dd0bcd911f783b23c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:52 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:53096
I20251210 14:57:52.932974 8963 tablet_replica.cc:333] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: stopping tablet replica
I20251210 14:57:52.933197 8963 raft_consensus.cc:2243] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:52.933398 8963 raft_consensus.cc:2272] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:52.934106 8963 ts_tablet_manager.cc:1916] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:52.934670 8964 tablet_replica.cc:333] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: stopping tablet replica
I20251210 14:57:52.934816 8964 raft_consensus.cc:2243] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:52.934927 8964 raft_consensus.cc:2272] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:52.935158 8965 tablet_replica.cc:333] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: stopping tablet replica
I20251210 14:57:52.935284 8965 raft_consensus.cc:2243] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:57:52.935544 8965 raft_consensus.cc:2272] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:52.936142 8965 ts_tablet_manager.cc:1916] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:52.936275 8964 ts_tablet_manager.cc:1916] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:52.936775 8963 ts_tablet_manager.cc:1929] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251210 14:57:52.936841 8963 log.cc:1199] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/73fafae580174dd0bcd911f783b23c98
I20251210 14:57:52.937101 8963 ts_tablet_manager.cc:1950] T 73fafae580174dd0bcd911f783b23c98 P 652a49856e4a43f886ed450c4189bffb: Deleting consensus metadata
I20251210 14:57:52.937803 8349 catalog_manager.cc:4985] TS 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533): tablet 73fafae580174dd0bcd911f783b23c98 (table test-table [id=d371400857224357b259b216ddbfec55]) successfully deleted
I20251210 14:57:52.938540 8964 ts_tablet_manager.cc:1929] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251210 14:57:52.938540 8965 ts_tablet_manager.cc:1929] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251210 14:57:52.938614 8965 log.cc:1199] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/73fafae580174dd0bcd911f783b23c98
I20251210 14:57:52.938614 8964 log.cc:1199] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/73fafae580174dd0bcd911f783b23c98
I20251210 14:57:52.938910 8964 ts_tablet_manager.cc:1950] T 73fafae580174dd0bcd911f783b23c98 P d93fedee92c5499da73abb9cf6254e03: Deleting consensus metadata
I20251210 14:57:52.938910 8965 ts_tablet_manager.cc:1950] T 73fafae580174dd0bcd911f783b23c98 P dbde94ed1791422b99fc4da5380764c9: Deleting consensus metadata
I20251210 14:57:52.939494 8349 catalog_manager.cc:4985] TS dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573): tablet 73fafae580174dd0bcd911f783b23c98 (table test-table [id=d371400857224357b259b216ddbfec55]) successfully deleted
I20251210 14:57:52.939615 8351 catalog_manager.cc:4985] TS d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775): tablet 73fafae580174dd0bcd911f783b23c98 (table test-table [id=d371400857224357b259b216ddbfec55]) successfully deleted
Dec 10 14:57:52 dist-test-slave-nb51 krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 10 14:57:52 dist-test-slave-nb51 krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:52.959098 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:52.951663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.6.115.126:45499)
1210 14:57:52.951938 (+ 275us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:52.951942 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:52.951958 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:57:52.951996 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:52.951998 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:52.952042 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:52.952119 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:52.952866 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:52.953492 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:52.954156 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:52.954299 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:52.956350 (+ 2051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:52.956367 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:52.956370 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:52.956393 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:52.957969 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:52.958307 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:52.958309 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:52.958310 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:52.958346 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:52.958493 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:52.958495 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:52.958497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:52.958628 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:52.958718 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:52.958874 (+ 156us) server_negotiation.cc:300] Negotiation successful
1210 14:57:52.958967 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
I20251210 14:57:52.961326 8364 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:52.961802 8364 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:52.966408 8645 tablet_service.cc:1505] Processing CreateTablet for tablet 1ff5184ef74a4b828fc37f637f86472f (DEFAULT_TABLE table=test-table [id=947fa444306b4cb6ab84140e25492909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:52.966429 8503 tablet_service.cc:1505] Processing CreateTablet for tablet 1ff5184ef74a4b828fc37f637f86472f (DEFAULT_TABLE table=test-table [id=947fa444306b4cb6ab84140e25492909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:52.966728 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ff5184ef74a4b828fc37f637f86472f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:52.966749 8503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ff5184ef74a4b828fc37f637f86472f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:52.968618 8975 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Bootstrap starting.
I20251210 14:57:52.968832 8787 tablet_service.cc:1505] Processing CreateTablet for tablet 1ff5184ef74a4b828fc37f637f86472f (DEFAULT_TABLE table=test-table [id=947fa444306b4cb6ab84140e25492909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:52.969127 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ff5184ef74a4b828fc37f637f86472f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:52.969224 8974 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Bootstrap starting.
I20251210 14:57:52.969628 8975 tablet_bootstrap.cc:654] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:52.970101 8974 tablet_bootstrap.cc:654] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:52.970609 8975 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: No bootstrap required, opened a new log
I20251210 14:57:52.970690 8975 ts_tablet_manager.cc:1403] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251210 14:57:52.971110 8975 raft_consensus.cc:359] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.971196 8975 raft_consensus.cc:385] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:52.971225 8975 raft_consensus.cc:740] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Initialized, Role: FOLLOWER
I20251210 14:57:52.971354 8975 consensus_queue.cc:260] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.971596 8978 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Bootstrap starting.
I20251210 14:57:52.971738 8975 ts_tablet_manager.cc:1434] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:52.971395 8974 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: No bootstrap required, opened a new log
I20251210 14:57:52.972043 8974 ts_tablet_manager.cc:1403] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:57:52.972544 8978 tablet_bootstrap.cc:654] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:52.972482 8974 raft_consensus.cc:359] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.972595 8974 raft_consensus.cc:385] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:52.972654 8974 raft_consensus.cc:740] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Initialized, Role: FOLLOWER
I20251210 14:57:52.972818 8974 consensus_queue.cc:260] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.973062 8974 ts_tablet_manager.cc:1434] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:52.974028 8978 tablet_bootstrap.cc:492] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: No bootstrap required, opened a new log
I20251210 14:57:52.974156 8978 ts_tablet_manager.cc:1403] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:57:52.974618 8978 raft_consensus.cc:359] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.974793 8978 raft_consensus.cc:385] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:52.974845 8978 raft_consensus.cc:740] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Initialized, Role: FOLLOWER
I20251210 14:57:52.975018 8978 consensus_queue.cc:260] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:52.975298 8978 ts_tablet_manager.cc:1434] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:53.211527 8956 raft_consensus.cc:493] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:53.211696 8956 raft_consensus.cc:515] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.212186 8956 leader_election.cc:290] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:53.212594 8522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ff5184ef74a4b828fc37f637f86472f" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb" is_pre_election: true
I20251210 14:57:53.212627 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ff5184ef74a4b828fc37f637f86472f" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03" is_pre_election: true
I20251210 14:57:53.212725 8522 raft_consensus.cc:2468] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 0.
I20251210 14:57:53.212819 8798 raft_consensus.cc:2468] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 0.
I20251210 14:57:53.213061 8597 leader_election.cc:304] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:53.213233 8956 raft_consensus.cc:2804] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:53.213303 8956 raft_consensus.cc:493] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:53.213330 8956 raft_consensus.cc:3060] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.214284 8956 raft_consensus.cc:515] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.214584 8956 leader_election.cc:290] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 election: Requested vote from peers 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:53.214962 8798 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ff5184ef74a4b828fc37f637f86472f" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03"
I20251210 14:57:53.214955 8522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ff5184ef74a4b828fc37f637f86472f" candidate_uuid: "dbde94ed1791422b99fc4da5380764c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a49856e4a43f886ed450c4189bffb"
I20251210 14:57:53.215076 8798 raft_consensus.cc:3060] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.215090 8522 raft_consensus.cc:3060] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.216173 8522 raft_consensus.cc:2468] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 1.
I20251210 14:57:53.216173 8798 raft_consensus.cc:2468] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbde94ed1791422b99fc4da5380764c9 in term 1.
I20251210 14:57:53.216454 8599 leader_election.cc:304] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d93fedee92c5499da73abb9cf6254e03, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:53.216630 8956 raft_consensus.cc:2804] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:53.216727 8956 raft_consensus.cc:697] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 1 LEADER]: Becoming Leader. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Running, Role: LEADER
I20251210 14:57:53.216862 8956 consensus_queue.cc:237] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.218696 8362 catalog_manager.cc:5654] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbde94ed1791422b99fc4da5380764c9 (127.6.115.66). New cstate: current_term: 1 leader_uuid: "dbde94ed1791422b99fc4da5380764c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } health_report { overall_health: UNKNOWN } } }
I20251210 14:57:53.265199 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:53.262529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.6.115.66:39573)
1210 14:57:53.262831 (+ 302us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:53.262835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:53.262847 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:57:53.262905 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:53.262908 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:53.262955 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:53.263034 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:53.263439 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:53.263929 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:53.264571 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:53.264730 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:53.264768 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:53.264826 (+ 58us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:53.264887 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:53.264982 (+ 95us) server_negotiation.cc:300] Negotiation successful
1210 14:57:53.265021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":130,"threads_started":1}
I20251210 14:57:53.271286 8798 raft_consensus.cc:1275] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Refusing update from remote peer dbde94ed1791422b99fc4da5380764c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:53.271327 8522 raft_consensus.cc:1275] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Refusing update from remote peer dbde94ed1791422b99fc4da5380764c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:53.271734 8882 consensus_queue.cc:1048] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:53.271879 8956 consensus_queue.cc:1048] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:53.302793 8364 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:53.302942 8364 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:53.304836 8364 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Sending DeleteTablet for 3 replicas of tablet 1ff5184ef74a4b828fc37f637f86472f
I20251210 14:57:53.305348 8503 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ff5184ef74a4b828fc37f637f86472f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:53 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:45678
I20251210 14:57:53.305408 8645 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ff5184ef74a4b828fc37f637f86472f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:53 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:53096
I20251210 14:57:53.305447 8787 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ff5184ef74a4b828fc37f637f86472f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:53 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:34586
I20251210 14:57:53.305590 8963 tablet_replica.cc:333] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: stopping tablet replica
I20251210 14:57:53.305616 8965 tablet_replica.cc:333] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: stopping tablet replica
I20251210 14:57:53.305711 8963 raft_consensus.cc:2243] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:53.305742 8965 raft_consensus.cc:2243] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:57:53.305830 8963 raft_consensus.cc:2272] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:53.305917 8965 raft_consensus.cc:2272] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:53.306093 8964 tablet_replica.cc:333] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: stopping tablet replica
I20251210 14:57:53.306205 8964 raft_consensus.cc:2243] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:53.306296 8964 raft_consensus.cc:2272] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:53.306576 8963 ts_tablet_manager.cc:1916] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:53.306975 8965 ts_tablet_manager.cc:1916] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:53.307164 8364 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:57:53.307569 8364 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:57:53.308923 8963 ts_tablet_manager.cc:1929] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:53.309002 8963 log.cc:1199] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/1ff5184ef74a4b828fc37f637f86472f
I20251210 14:57:53.309270 8963 ts_tablet_manager.cc:1950] T 1ff5184ef74a4b828fc37f637f86472f P 652a49856e4a43f886ed450c4189bffb: Deleting consensus metadata
I20251210 14:57:53.309389 8965 ts_tablet_manager.cc:1929] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:53.309453 8965 log.cc:1199] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/1ff5184ef74a4b828fc37f637f86472f
I20251210 14:57:53.309727 8965 ts_tablet_manager.cc:1950] T 1ff5184ef74a4b828fc37f637f86472f P dbde94ed1791422b99fc4da5380764c9: Deleting consensus metadata
I20251210 14:57:53.310000 8349 catalog_manager.cc:4985] TS 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533): tablet 1ff5184ef74a4b828fc37f637f86472f (table test-table [id=947fa444306b4cb6ab84140e25492909]) successfully deleted
I20251210 14:57:53.310343 8964 ts_tablet_manager.cc:1916] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:53.310906 8349 catalog_manager.cc:4985] TS dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573): tablet 1ff5184ef74a4b828fc37f637f86472f (table test-table [id=947fa444306b4cb6ab84140e25492909]) successfully deleted
I20251210 14:57:53.312449 8503 tablet_service.cc:1505] Processing CreateTablet for tablet 909ecbfa331249babba93c072e5dbe85 (DEFAULT_TABLE table=test-table [id=cfe8ec4033e54788ba7b91a1da108311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:53.312508 8645 tablet_service.cc:1505] Processing CreateTablet for tablet 909ecbfa331249babba93c072e5dbe85 (DEFAULT_TABLE table=test-table [id=cfe8ec4033e54788ba7b91a1da108311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:53.312584 8964 ts_tablet_manager.cc:1929] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:57:53.312647 8964 log.cc:1199] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/1ff5184ef74a4b828fc37f637f86472f
I20251210 14:57:53.312693 8503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909ecbfa331249babba93c072e5dbe85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:53.312693 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909ecbfa331249babba93c072e5dbe85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:53.312801 8787 tablet_service.cc:1505] Processing CreateTablet for tablet 909ecbfa331249babba93c072e5dbe85 (DEFAULT_TABLE table=test-table [id=cfe8ec4033e54788ba7b91a1da108311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:57:53.312875 8964 ts_tablet_manager.cc:1950] T 1ff5184ef74a4b828fc37f637f86472f P d93fedee92c5499da73abb9cf6254e03: Deleting consensus metadata
I20251210 14:57:53.313014 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909ecbfa331249babba93c072e5dbe85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:53.314678 8978 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Bootstrap starting.
I20251210 14:57:53.314743 8975 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb: Bootstrap starting.
I20251210 14:57:53.314806 8974 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Bootstrap starting.
I20251210 14:57:53.315548 8978 tablet_bootstrap.cc:654] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:53.315548 8975 tablet_bootstrap.cc:654] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:53.315698 8974 tablet_bootstrap.cc:654] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:53.316154 8351 catalog_manager.cc:4985] TS d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775): tablet 1ff5184ef74a4b828fc37f637f86472f (table test-table [id=947fa444306b4cb6ab84140e25492909]) successfully deleted
I20251210 14:57:53.316752 8975 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb: No bootstrap required, opened a new log
I20251210 14:57:53.316833 8978 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: No bootstrap required, opened a new log
I20251210 14:57:53.316839 8975 ts_tablet_manager.cc:1403] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251210 14:57:53.316918 8978 ts_tablet_manager.cc:1403] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251210 14:57:53.317268 8975 raft_consensus.cc:359] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.317303 8978 raft_consensus.cc:359] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.317396 8975 raft_consensus.cc:385] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:53.317409 8978 raft_consensus.cc:385] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:53.317433 8975 raft_consensus.cc:740] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Initialized, Role: FOLLOWER
I20251210 14:57:53.317443 8978 raft_consensus.cc:740] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93fedee92c5499da73abb9cf6254e03, State: Initialized, Role: FOLLOWER
I20251210 14:57:53.317548 8975 consensus_queue.cc:260] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.317548 8978 consensus_queue.cc:260] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.317780 8978 ts_tablet_manager.cc:1434] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:57:53.317780 8975 ts_tablet_manager.cc:1434] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251210 14:57:53.322080 8974 tablet_bootstrap.cc:492] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: No bootstrap required, opened a new log
I20251210 14:57:53.322166 8974 ts_tablet_manager.cc:1403] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251210 14:57:53.322520 8974 raft_consensus.cc:359] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.322597 8974 raft_consensus.cc:385] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:53.322618 8974 raft_consensus.cc:740] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde94ed1791422b99fc4da5380764c9, State: Initialized, Role: FOLLOWER
I20251210 14:57:53.322726 8974 consensus_queue.cc:260] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.322944 8974 ts_tablet_manager.cc:1434] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251210 14:57:53.490502 8979 raft_consensus.cc:493] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:57:53.490656 8979 raft_consensus.cc:515] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.491051 8979 leader_election.cc:290] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:53.491569 8664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "909ecbfa331249babba93c072e5dbe85" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9" is_pre_election: true
I20251210 14:57:53.491569 8807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "909ecbfa331249babba93c072e5dbe85" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03" is_pre_election: true
I20251210 14:57:53.491717 8664 raft_consensus.cc:2468] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 0.
I20251210 14:57:53.491717 8807 raft_consensus.cc:2468] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 0.
I20251210 14:57:53.492025 8457 leader_election.cc:304] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, d93fedee92c5499da73abb9cf6254e03; no voters:
I20251210 14:57:53.492187 8979 raft_consensus.cc:2804] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:57:53.492254 8979 raft_consensus.cc:493] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:57:53.492290 8979 raft_consensus.cc:3060] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.493252 8979 raft_consensus.cc:515] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.493605 8979 leader_election.cc:290] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 election: Requested vote from peers dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573), d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775)
I20251210 14:57:53.493955 8807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "909ecbfa331249babba93c072e5dbe85" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93fedee92c5499da73abb9cf6254e03"
I20251210 14:57:53.493955 8664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "909ecbfa331249babba93c072e5dbe85" candidate_uuid: "652a49856e4a43f886ed450c4189bffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbde94ed1791422b99fc4da5380764c9"
I20251210 14:57:53.494077 8664 raft_consensus.cc:3060] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.494077 8807 raft_consensus.cc:3060] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:53.495138 8807 raft_consensus.cc:2468] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 1.
I20251210 14:57:53.495138 8664 raft_consensus.cc:2468] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a49856e4a43f886ed450c4189bffb in term 1.
I20251210 14:57:53.495446 8455 leader_election.cc:304] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a49856e4a43f886ed450c4189bffb, dbde94ed1791422b99fc4da5380764c9; no voters:
I20251210 14:57:53.495628 8979 raft_consensus.cc:2804] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:53.495795 8979 raft_consensus.cc:697] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [term 1 LEADER]: Becoming Leader. State: Replica: 652a49856e4a43f886ed450c4189bffb, State: Running, Role: LEADER
I20251210 14:57:53.495935 8979 consensus_queue.cc:237] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } }
I20251210 14:57:53.497658 8362 catalog_manager.cc:5654] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a49856e4a43f886ed450c4189bffb (127.6.115.65). New cstate: current_term: 1 leader_uuid: "652a49856e4a43f886ed450c4189bffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a49856e4a43f886ed450c4189bffb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 } health_report { overall_health: UNKNOWN } } }
I20251210 14:57:53.511564 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:53.508857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.6.115.65:35533)
1210 14:57:53.509173 (+ 316us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:53.509176 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:53.509188 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:57:53.509223 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:53.509225 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:53.509265 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:53.509334 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:57:53.509727 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:53.510214 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:53.510786 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:53.510961 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:53.511002 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:57:53.511064 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:57:53.511124 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:53.511320 (+ 196us) server_negotiation.cc:300] Negotiation successful
1210 14:57:53.511378 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":155,"threads_started":1}
I20251210 14:57:53.517153 8664 raft_consensus.cc:1275] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Refusing update from remote peer 652a49856e4a43f886ed450c4189bffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:53.517266 8807 raft_consensus.cc:1275] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Refusing update from remote peer 652a49856e4a43f886ed450c4189bffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:57:53.517537 8991 consensus_queue.cc:1048] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbde94ed1791422b99fc4da5380764c9" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:57:53.517688 8979 consensus_queue.cc:1048] T 909ecbfa331249babba93c072e5dbe85 P 652a49856e4a43f886ed450c4189bffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93fedee92c5499da73abb9cf6254e03" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251210 14:57:53.531860 8946 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251210 14:57:55.083952 8362 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:55.084088 8362 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:57:55.086318 8362 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 5e0de6dc53cd4126ae23a51ea7bf0724: Sending DeleteTablet for 3 replicas of tablet 909ecbfa331249babba93c072e5dbe85
I20251210 14:57:55.086865 8503 tablet_service.cc:1552] Processing DeleteTablet for tablet 909ecbfa331249babba93c072e5dbe85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:45678
I20251210 14:57:55.086853 8645 tablet_service.cc:1552] Processing DeleteTablet for tablet 909ecbfa331249babba93c072e5dbe85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:53096
I20251210 14:57:55.086882 8787 tablet_service.cc:1552] Processing DeleteTablet for tablet 909ecbfa331249babba93c072e5dbe85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:34586
I20251210 14:57:55.087198 9020 tablet_replica.cc:333] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: stopping tablet replica
I20251210 14:57:55.087226 9022 tablet_replica.cc:333] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: stopping tablet replica
I20251210 14:57:55.087327 9020 raft_consensus.cc:2243] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:55.087347 9022 raft_consensus.cc:2243] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:57:55.087440 9020 raft_consensus.cc:2272] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:55.087446 9022 raft_consensus.cc:2272] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:57:55.087518 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 8438
I20251210 14:57:55.088972 9022 ts_tablet_manager.cc:1916] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:55.089188 9020 ts_tablet_manager.cc:1916] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:57:55.091158 9022 ts_tablet_manager.cc:1929] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251210 14:57:55.091231 9022 log.cc:1199] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/909ecbfa331249babba93c072e5dbe85
I20251210 14:57:55.091529 9022 ts_tablet_manager.cc:1950] T 909ecbfa331249babba93c072e5dbe85 P d93fedee92c5499da73abb9cf6254e03: Deleting consensus metadata
I20251210 14:57:55.092105 8351 catalog_manager.cc:4985] TS d93fedee92c5499da73abb9cf6254e03 (127.6.115.67:39775): tablet 909ecbfa331249babba93c072e5dbe85 (table test-table [id=cfe8ec4033e54788ba7b91a1da108311]) successfully deleted
I20251210 14:57:55.092358 9020 ts_tablet_manager.cc:1929] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251210 14:57:55.092438 9020 log.cc:1199] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/909ecbfa331249babba93c072e5dbe85
I20251210 14:57:55.092728 9020 ts_tablet_manager.cc:1950] T 909ecbfa331249babba93c072e5dbe85 P dbde94ed1791422b99fc4da5380764c9: Deleting consensus metadata
I20251210 14:57:55.093480 8349 catalog_manager.cc:4985] TS dbde94ed1791422b99fc4da5380764c9 (127.6.115.66:39573): tablet 909ecbfa331249babba93c072e5dbe85 (table test-table [id=cfe8ec4033e54788ba7b91a1da108311]) successfully deleted
W20251210 14:57:55.096110 8349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:35533 (error 108)
I20251210 14:57:55.096300 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 8580
I20251210 14:57:55.097731 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:55.097144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35533 (local address 127.0.0.1:45704)
1210 14:57:55.097525 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:55.097591 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:35533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":66,"threads_started":1}
W20251210 14:57:55.097847 8349 catalog_manager.cc:4712] TS 652a49856e4a43f886ed450c4189bffb (127.6.115.65:35533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 909ecbfa331249babba93c072e5dbe85: Network error: Client connection negotiation failed: client connection to 127.6.115.65:35533: connect: Connection refused (error 111)
I20251210 14:57:55.103348 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 8722
I20251210 14:57:55.109788 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 8332
2025-12-10T14:57:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7307 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:55 dist-test-slave-nb51 krb5kdc[9027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:55 dist-test-slave-nb51 krb5kdc[9027](info): set up 2 sockets
Dec 10 14:57:55 dist-test-slave-nb51 krb5kdc[9027](info): commencing operation
krb5kdc: starting...
W20251210 14:57:57.077574 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:57:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:57.205017 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:36437
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:57.304140 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:57.304347 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:57.304437 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:57.307478 9043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:57:57.307533 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:57.307570 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:57.307603 9043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:57:57.307641 9043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:57:57.311568 9043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:57.312501 9043 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:57.313326 9043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:57.318735 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:57.318727 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:57.318722 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:57.318804 9043 server_base.cc:1047] running on GCE node
I20251210 14:57:57.319406 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:57.320256 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:57.321424 9043 hybrid_clock.cc:648] HybridClock initialized: now 1765378677321413 us; error 24 us; skew 500 ppm
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:57.323778 9043 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:57:57.324712 9043 webserver.cc:492] Webserver started at http://127.6.115.126:39655/ using document root <none> and password file <none>
I20251210 14:57:57.325171 9043 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:57.325213 9043 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:57.325500 9043 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:57.327119 9043 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "efebcc8f6281339142ff6dabf6c8cca6"
server_key_iv: "a2d318e96de44362d3abf711375a3404"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.327587 9043 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "efebcc8f6281339142ff6dabf6c8cca6"
server_key_iv: "a2d318e96de44362d3abf711375a3404"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.330641 9043 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:57:57.332576 9058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.333582 9043 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:57:57.333673 9043 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "efebcc8f6281339142ff6dabf6c8cca6"
server_key_iv: "a2d318e96de44362d3abf711375a3404"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.333745 9043 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:57.348531 9043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:57.351447 9043 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:57.351593 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:57.358557 9043 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:36437
I20251210 14:57:57.358559 9110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:36437 every 8 connection(s)
I20251210 14:57:57.359515 9043 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:57:57.362112 9111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:57:57.367312 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2: Bootstrap starting.
I20251210 14:57:57.369474 9111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2: Neither blocks nor log segments found. Creating new log.
I20251210 14:57:57.369771 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9043
I20251210 14:57:57.369895 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:57:57.370142 6605 external_mini_cluster.cc:1468] Setting key c5c1e6a548ab19bb68d54781dce2e68c
I20251210 14:57:57.370327 9111 log.cc:826] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2: Log is configured to *not* fsync() on all Append() calls
I20251210 14:57:57.372318 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2: No bootstrap required, opened a new log
I20251210 14:57:57.375284 9111 raft_consensus.cc:359] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } }
I20251210 14:57:57.375447 9111 raft_consensus.cc:385] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:57:57.375491 9111 raft_consensus.cc:740] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c26d5d13b64ad5bd8102bc4e1341d2, State: Initialized, Role: FOLLOWER
I20251210 14:57:57.375939 9111 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } }
I20251210 14:57:57.376128 9111 raft_consensus.cc:399] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:57:57.376194 9111 raft_consensus.cc:493] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:57:57.376281 9111 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:57:57.377034 9111 raft_consensus.cc:515] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } }
I20251210 14:57:57.377395 9111 leader_election.cc:304] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c26d5d13b64ad5bd8102bc4e1341d2; no voters:
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:57.377722 9111 leader_election.cc:290] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:57:57.377830 9116 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:57:57.378021 9116 raft_consensus.cc:697] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [term 1 LEADER]: Becoming Leader. State: Replica: a8c26d5d13b64ad5bd8102bc4e1341d2, State: Running, Role: LEADER
I20251210 14:57:57.378335 9116 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } }
I20251210 14:57:57.378842 9111 sys_catalog.cc:565] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:57:57.380183 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c26d5d13b64ad5bd8102bc4e1341d2. Latest consensus state: current_term: 1 leader_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } } }
I20251210 14:57:57.380190 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c26d5d13b64ad5bd8102bc4e1341d2" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 36437 } } }
I20251210 14:57:57.380344 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:57.380349 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2 [sys.catalog]: This master's current role is: LEADER
I20251210 14:57:57.380812 9125 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:57:57.384052 9125 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:57:57.385141 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:57.371118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.6.115.126:36437)
1210 14:57:57.371501 (+ 383us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:57.371512 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:57.371552 (+ 40us) server_negotiation.cc:408] Connection header received
1210 14:57:57.372198 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:57.372222 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:57.372589 (+ 367us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:57.372927 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:57.373787 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.374953 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:57.375679 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.375938 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:57.378281 (+ 2343us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:57.378300 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:57.378312 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:57.378348 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:57.381745 (+ 3397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.382259 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.382266 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.382271 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.382359 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.382630 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.382633 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.382635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.383005 (+ 370us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:57.383204 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:57.384003 (+ 799us) server_negotiation.cc:300] Negotiation successful
1210 14:57:57.384247 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20251210 14:57:57.389760 9125 catalog_manager.cc:1357] Generated new cluster ID: 0b0e457ff3304d7bb567cb25d100bc63
I20251210 14:57:57.389822 9125 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:57:57.411156 9125 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:57:57.412305 9125 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:57:57.421651 9125 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a8c26d5d13b64ad5bd8102bc4e1341d2: Generated new TSK 0
I20251210 14:57:57.422399 9125 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:57.474429 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36437
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:57.576478 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:57.576706 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:57.576798 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:57.579795 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:57.579855 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:57.579962 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:57:57.584182 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:57.585215 9139 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:57.586071 9139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:57.592293 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:57.592233 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:57.592347 9139 server_base.cc:1047] running on GCE node
W20251210 14:57:57.592231 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:57.592728 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:57.593261 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:57.594424 9139 hybrid_clock.cc:648] HybridClock initialized: now 1765378677594417 us; error 24 us; skew 500 ppm
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:57.596920 9139 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:57:57.597908 9139 webserver.cc:492] Webserver started at http://127.6.115.65:37017/ using document root <none> and password file <none>
I20251210 14:57:57.598400 9139 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:57.598469 9139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:57.598652 9139 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:57.600281 9139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "b754d8d9a61c4369928cf86ae44d3ade"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "e5b30c39dfa8e07babe806ca71855a56"
server_key_iv: "73b72c266989a06197f270b002c452e1"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.600723 9139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "b754d8d9a61c4369928cf86ae44d3ade"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "e5b30c39dfa8e07babe806ca71855a56"
server_key_iv: "73b72c266989a06197f270b002c452e1"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.604024 9139 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:57.606141 9154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.607054 9139 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:57.607180 9139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "b754d8d9a61c4369928cf86ae44d3ade"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "e5b30c39dfa8e07babe806ca71855a56"
server_key_iv: "73b72c266989a06197f270b002c452e1"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.607257 9139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:57.623032 9139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:57.625859 9139 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:57.626029 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:57.626519 9139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:57.627331 9139 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:57.627380 9139 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.627441 9139 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:57.627475 9139 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.636350 9139 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:35983
I20251210 14:57:57.636399 9267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:35983 every 8 connection(s)
I20251210 14:57:57.637236 9139 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:57:57.639637 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9139
I20251210 14:57:57.639751 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:57:57.640022 6605 external_mini_cluster.cc:1468] Setting key cf992613f582ca5181c22ce05baf707c
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:57:57.650776 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:57.638964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:34501 (local address 127.6.115.126:36437)
1210 14:57:57.639090 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:57.639094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:57.639800 (+ 706us) server_negotiation.cc:408] Connection header received
1210 14:57:57.640864 (+ 1064us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:57.640868 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:57.640921 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:57.640999 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:57.643168 (+ 2169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.644009 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:57.644688 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.644925 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:57.647346 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:57.647363 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:57.647365 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:57.647389 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:57.648829 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.649378 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.649384 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.649388 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.649436 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.649697 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.649700 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.649703 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.649861 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:57.649949 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:57.650475 (+ 526us) server_negotiation.cc:300] Negotiation successful
1210 14:57:57.650612 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:57:57.651365 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:57.639185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36437 (local address 127.6.115.65:34501)
1210 14:57:57.639654 (+ 469us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:57.639699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:57.640670 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:57.641183 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:57.641195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:57.641742 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:57.642983 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:57.643003 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.644148 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:57.644150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:57.644565 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:57.644572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.644740 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:57.645398 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:57.645430 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:57.647174 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:57.648966 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:57.648972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:57.648983 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:57.649264 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:57.649527 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:57.649530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:57.649531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:57.649613 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:57.649959 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:57.649967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:57.650251 (+ 284us) client_negotiation.cc:769] Sending connection context
1210 14:57:57.650438 (+ 187us) client_negotiation.cc:240] Negotiation successful
1210 14:57:57.650706 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:57:57.652668 9268 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36437
I20251210 14:57:57.652913 9268 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:57.653574 9268 heartbeater.cc:507] Master 127.6.115.126:36437 requested a full tablet report, sending...
I20251210 14:57:57.655079 9075 ts_manager.cc:194] Registered new tserver with Master: b754d8d9a61c4369928cf86ae44d3ade (127.6.115.65:35983)
I20251210 14:57:57.656270 9075 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:34501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:57.689750 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36437
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:57.791556 9275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:57.791751 9275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:57.791862 9275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:57.795133 9275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:57.795215 9275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:57.795320 9275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:57:57.799253 9275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:57.800215 9275 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:57.801246 9275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:57:57.808092 9280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:57.808182 9275 server_base.cc:1047] running on GCE node
W20251210 14:57:57.808118 9283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:57.808110 9281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:57.808598 9275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:57.809114 9275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:57.810283 9275 hybrid_clock.cc:648] HybridClock initialized: now 1765378677810268 us; error 31 us; skew 500 ppm
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:57.812780 9275 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:57:57.813798 9275 webserver.cc:492] Webserver started at http://127.6.115.66:44445/ using document root <none> and password file <none>
I20251210 14:57:57.814354 9275 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:57.814420 9275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:57.814594 9275 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:57.816182 9275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "2268591576be4f44ad07d00a23b9e520"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "75815ce1302dfc3fbf60e83d202db277"
server_key_iv: "a56ad89e5d7956545dd358fb98f16ece"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.816623 9275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "2268591576be4f44ad07d00a23b9e520"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "75815ce1302dfc3fbf60e83d202db277"
server_key_iv: "a56ad89e5d7956545dd358fb98f16ece"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.819804 9275 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251210 14:57:57.821883 9290 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.822887 9275 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251210 14:57:57.822992 9275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "2268591576be4f44ad07d00a23b9e520"
format_stamp: "Formatted at 2025-12-10 14:57:57 on dist-test-slave-nb51"
server_key: "75815ce1302dfc3fbf60e83d202db277"
server_key_iv: "a56ad89e5d7956545dd358fb98f16ece"
server_key_version: "encryptionkey@0"
I20251210 14:57:57.823082 9275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:57.853098 9275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:57.856040 9275 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:57.856230 9275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:57.856771 9275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:57.857646 9275 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:57.857697 9275 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.857753 9275 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:57.857784 9275 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:57.866496 9275 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:36061
I20251210 14:57:57.866530 9403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:36061 every 8 connection(s)
I20251210 14:57:57.867403 9275 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:57:57.875033 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9275
I20251210 14:57:57.875159 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:57:57.875442 6605 external_mini_cluster.cc:1468] Setting key 5fab76cb1a07d615954ac2170a07985d
Dec 10 14:57:57 dist-test-slave-nb51 krb5kdc[9027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:57.879210 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:57.869000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:47645 (local address 127.6.115.126:36437)
1210 14:57:57.869126 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:57.869129 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:57.870011 (+ 882us) server_negotiation.cc:408] Connection header received
1210 14:57:57.870782 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:57.870785 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:57.870833 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:57.870926 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:57.872536 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.873032 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:57.873675 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.873800 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:57.876053 (+ 2253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:57.876069 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:57.876071 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:57.876094 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:57.877498 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.877920 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.877924 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.877926 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.877970 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:57.878222 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:57.878226 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:57.878228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:57.878382 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:57.878470 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:57.878924 (+ 454us) server_negotiation.cc:300] Negotiation successful
1210 14:57:57.879041 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251210 14:57:57.879896 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:57.869393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36437 (local address 127.6.115.66:47645)
1210 14:57:57.869840 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:57.869874 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:57.870602 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:57.871060 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:57.871067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:57.871510 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:57.872398 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:57.872412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.873128 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:57.873130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:57.873591 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:57.873599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:57.873762 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:57.874333 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:57.874356 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:57.875892 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:57.877599 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:57.877604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:57.877614 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:57.877831 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:57.878047 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:57.878050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:57.878054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:57.878134 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:57.878485 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:57.878494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:57.878703 (+ 209us) client_negotiation.cc:769] Sending connection context
1210 14:57:57.878880 (+ 177us) client_negotiation.cc:240] Negotiation successful
1210 14:57:57.879129 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20251210 14:57:57.881068 9404 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36437
I20251210 14:57:57.881290 9404 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:57.881776 9404 heartbeater.cc:507] Master 127.6.115.126:36437 requested a full tablet report, sending...
I20251210 14:57:57.882781 9075 ts_manager.cc:194] Registered new tserver with Master: 2268591576be4f44ad07d00a23b9e520 (127.6.115.66:36061)
I20251210 14:57:57.883582 9075 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:47645
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:57:57.920679 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:36437
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:57:58.023012 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:57:58.023223 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:57:58.023281 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:57:58.026516 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:57:58.026574 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:57:58.026664 9411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:57:58.030795 9411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:57:58.031774 9411 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:57:58.032617 9411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251210 14:57:58.039263 9411 server_base.cc:1047] running on GCE node
W20251210 14:57:58.039187 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:58.039283 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:57:58.039188 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:57:58.039880 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:57:58.040429 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:57:58.041564 9411 hybrid_clock.cc:648] HybridClock initialized: now 1765378678041551 us; error 38 us; skew 500 ppm
Dec 10 14:57:58 dist-test-slave-nb51 krb5kdc[9027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:57:58.044226 9411 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:57:58.045204 9411 webserver.cc:492] Webserver started at http://127.6.115.67:34297/ using document root <none> and password file <none>
I20251210 14:57:58.045768 9411 fs_manager.cc:362] Metadata directory not provided
I20251210 14:57:58.045825 9411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:57:58.045996 9411 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:57:58.047631 9411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "ee6cc9d7f621404fab34016542dee31d"
format_stamp: "Formatted at 2025-12-10 14:57:58 on dist-test-slave-nb51"
server_key: "dd423c654c789213795d016742457839"
server_key_iv: "0e14b24b55413ecd46770a992630a3bc"
server_key_version: "encryptionkey@0"
I20251210 14:57:58.048089 9411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "ee6cc9d7f621404fab34016542dee31d"
format_stamp: "Formatted at 2025-12-10 14:57:58 on dist-test-slave-nb51"
server_key: "dd423c654c789213795d016742457839"
server_key_iv: "0e14b24b55413ecd46770a992630a3bc"
server_key_version: "encryptionkey@0"
I20251210 14:57:58.051359 9411 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:57:58.053556 9426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:58.054533 9411 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:57:58.054636 9411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "ee6cc9d7f621404fab34016542dee31d"
format_stamp: "Formatted at 2025-12-10 14:57:58 on dist-test-slave-nb51"
server_key: "dd423c654c789213795d016742457839"
server_key_iv: "0e14b24b55413ecd46770a992630a3bc"
server_key_version: "encryptionkey@0"
I20251210 14:57:58.054725 9411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:57:58.064616 9411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:57:58.067410 9411 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:57:58.067576 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:57:58.068084 9411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:57:58.068890 9411 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:57:58.068951 9411 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:58.069020 9411 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:57:58.069067 9411 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:57:58.077615 9411 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:39531
I20251210 14:57:58.077639 9539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:39531 every 8 connection(s)
I20251210 14:57:58.078536 9411 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:57:58.085929 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9411
I20251210 14:57:58.086022 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:57:58.086251 6605 external_mini_cluster.cc:1468] Setting key f768164f6652b83953772b4d686f5213
Dec 10 14:57:58 dist-test-slave-nb51 krb5kdc[9027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:57:58.090297 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:58.080193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:50709 (local address 127.6.115.126:36437)
1210 14:57:58.080340 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:58.080346 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:58.081034 (+ 688us) server_negotiation.cc:408] Connection header received
1210 14:57:58.081869 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:58.081872 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:58.081920 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:58.081998 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:58.083600 (+ 1602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:58.084078 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:57:58.084660 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:57:58.084789 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:58.087006 (+ 2217us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:57:58.087030 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:57:58.087032 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:57:58.087055 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:57:58.088508 (+ 1453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:58.088956 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:58.088960 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:58.088961 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:58.089002 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:57:58.089263 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:57:58.089267 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:57:58.089269 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:57:58.089456 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:57:58.089541 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1210 14:57:58.090006 (+ 465us) server_negotiation.cc:300] Negotiation successful
1210 14:57:58.090139 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251210 14:57:58.090859 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:57:58.080479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:36437 (local address 127.6.115.67:50709)
1210 14:57:58.080877 (+ 398us) negotiation.cc:107] Waiting for socket to connect
1210 14:57:58.080913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1210 14:57:58.081678 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:57:58.082160 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:57:58.082167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:57:58.082560 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:57:58.083469 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:58.083481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:58.084169 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:57:58.084172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:57:58.084565 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:57:58.084571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:57:58.084739 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:57:58.085325 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:57:58.085345 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:57:58.086870 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:57:58.088621 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:58.088627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:58.088638 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:58.088859 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:58.089092 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:57:58.089094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:57:58.089096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:57:58.089178 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:57:58.089562 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:57:58.089569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:57:58.089780 (+ 211us) client_negotiation.cc:769] Sending connection context
1210 14:57:58.089962 (+ 182us) client_negotiation.cc:240] Negotiation successful
1210 14:57:58.090250 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20251210 14:57:58.091799 9540 heartbeater.cc:344] Connected to a master server at 127.6.115.126:36437
I20251210 14:57:58.092010 9540 heartbeater.cc:461] Registering TS with master...
I20251210 14:57:58.092481 9540 heartbeater.cc:507] Master 127.6.115.126:36437 requested a full tablet report, sending...
I20251210 14:57:58.093441 9075 ts_manager.cc:194] Registered new tserver with Master: ee6cc9d7f621404fab34016542dee31d (127.6.115.67:39531)
I20251210 14:57:58.093973 9075 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:50709
I20251210 14:57:58.099643 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251210 14:57:58.111119 9115 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:57:58.109797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.6.115.126:36437)
1210 14:57:58.109964 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1210 14:57:58.109971 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:57:58.110068 (+ 97us) server_negotiation.cc:408] Connection header received
1210 14:57:58.110215 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:57:58.110219 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:57:58.110268 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:57:58.110365 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:57:58.110994 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55096: BlockingRecv error: recv got EOF from 127.0.0.1:55096 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20251210 14:57:58.112594 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9139
I20251210 14:57:58.118288 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9275
I20251210 14:57:58.123587 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9411
I20251210 14:57:58.128881 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9043
2025-12-10T14:57:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3017 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:57:58 dist-test-slave-nb51 krb5kdc[9552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:57:58 dist-test-slave-nb51 krb5kdc[9552](info): set up 2 sockets
Dec 10 14:57:58 dist-test-slave-nb51 krb5kdc[9552](info): commencing operation
krb5kdc: starting...
W20251210 14:58:00.097791 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:00.223853 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:35809
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:35809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:00.324927 9568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:00.325145 9568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:00.325193 9568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:00.328299 9568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:00.328354 9568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:00.328377 9568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:00.328395 9568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:00.328413 9568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:00.332392 9568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:35809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:35809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:00.333343 9568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:00.334134 9568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:00.339304 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:00.339300 9576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:00.339304 9574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:00.339414 9568 server_base.cc:1047] running on GCE node
I20251210 14:58:00.340029 9568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:00.340828 9568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:00.342006 9568 hybrid_clock.cc:648] HybridClock initialized: now 1765378680341993 us; error 28 us; skew 500 ppm
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:00.344573 9568 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:00.345595 9568 webserver.cc:492] Webserver started at http://127.6.115.126:44487/ using document root <none> and password file <none>
I20251210 14:58:00.346096 9568 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:00.346148 9568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:00.346339 9568 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:00.347940 9568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "b1ff19e2eae04961af9964034f41bbec"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "ddd8421d81b4c8a771a9f6269ae0ca2b"
server_key_iv: "94193158ce93d2f516a43c3b8d2b700f"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.348421 9568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "b1ff19e2eae04961af9964034f41bbec"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "ddd8421d81b4c8a771a9f6269ae0ca2b"
server_key_iv: "94193158ce93d2f516a43c3b8d2b700f"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.351626 9568 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251210 14:58:00.353711 9583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.354626 9568 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251210 14:58:00.354733 9568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "b1ff19e2eae04961af9964034f41bbec"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "ddd8421d81b4c8a771a9f6269ae0ca2b"
server_key_iv: "94193158ce93d2f516a43c3b8d2b700f"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.354809 9568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:00.382615 9568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:00.388504 9568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:00.388708 9568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:00.396078 9568 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:35809
I20251210 14:58:00.396071 9635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:35809 every 8 connection(s)
I20251210 14:58:00.397233 9568 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:00.398947 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9568
I20251210 14:58:00.399089 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:00.399391 6605 external_mini_cluster.cc:1468] Setting key f7f26837ab9ee28d5b83dc0cb0cae001
I20251210 14:58:00.400628 9636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:00.406140 9636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec: Bootstrap starting.
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:00.408936 9636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:00.409862 9636 log.cc:826] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:00.412045 9636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec: No bootstrap required, opened a new log
I20251210 14:58:00.413160 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:00.400755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.6.115.126:35809)
1210 14:58:00.401160 (+ 405us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:00.401170 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:00.401205 (+ 35us) server_negotiation.cc:408] Connection header received
1210 14:58:00.402360 (+ 1155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:00.402379 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:00.402662 (+ 283us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:00.402965 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:00.403898 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.404652 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:00.405301 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.405623 (+ 322us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:00.408387 (+ 2764us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:00.408407 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:00.408421 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:00.408463 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:00.410643 (+ 2180us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.411172 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.411176 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.411180 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.411249 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.411503 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.411506 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.411507 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.411808 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:00.411968 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:00.412339 (+ 371us) server_negotiation.cc:300] Negotiation successful
1210 14:58:00.412589 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20251210 14:58:00.415182 9636 raft_consensus.cc:359] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } }
I20251210 14:58:00.415386 9636 raft_consensus.cc:385] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:00.415462 9636 raft_consensus.cc:740] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff19e2eae04961af9964034f41bbec, State: Initialized, Role: FOLLOWER
I20251210 14:58:00.415910 9636 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } }
I20251210 14:58:00.416050 9636 raft_consensus.cc:399] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:00.416126 9636 raft_consensus.cc:493] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:00.416229 9636 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:00.417114 9636 raft_consensus.cc:515] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } }
I20251210 14:58:00.417508 9636 leader_election.cc:304] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ff19e2eae04961af9964034f41bbec; no voters:
I20251210 14:58:00.417815 9636 leader_election.cc:290] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:00.417898 9641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:00.418123 9641 raft_consensus.cc:697] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [term 1 LEADER]: Becoming Leader. State: Replica: b1ff19e2eae04961af9964034f41bbec, State: Running, Role: LEADER
I20251210 14:58:00.418491 9641 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } }
I20251210 14:58:00.419050 9636 sys_catalog.cc:565] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:00.420208 9642 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1ff19e2eae04961af9964034f41bbec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } } }
I20251210 14:58:00.420351 9642 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:00.420346 9643 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ff19e2eae04961af9964034f41bbec. Latest consensus state: current_term: 1 leader_uuid: "b1ff19e2eae04961af9964034f41bbec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ff19e2eae04961af9964034f41bbec" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 35809 } } }
I20251210 14:58:00.420472 9643 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:00.420758 9650 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:00.423166 9650 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:00.427721 9650 catalog_manager.cc:1357] Generated new cluster ID: 86db5cc4ddc84a6ea1d581e0d0460d33
I20251210 14:58:00.427781 9650 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:00.438001 9650 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:00.439327 9650 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:00.446437 9650 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b1ff19e2eae04961af9964034f41bbec: Generated new TSK 0
I20251210 14:58:00.447276 9650 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:00.493958 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35809
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:00.595681 9664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:00.595873 9664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:00.595964 9664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:00.599179 9664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:00.599263 9664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:00.599385 9664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:00.603340 9664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:00.604305 9664 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:00.605069 9664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:00.611155 9670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:00.611148 9672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:00.611207 9664 server_base.cc:1047] running on GCE node
W20251210 14:58:00.611153 9669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:00.611630 9664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:00.612134 9664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:00.613281 9664 hybrid_clock.cc:648] HybridClock initialized: now 1765378680613253 us; error 41 us; skew 500 ppm
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:00.615573 9664 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:00.616431 9664 webserver.cc:492] Webserver started at http://127.6.115.65:44349/ using document root <none> and password file <none>
I20251210 14:58:00.616941 9664 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:00.616993 9664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:00.617158 9664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:00.618768 9664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "f05b45999d234d77848e50f7c8f0e15e"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "47605130408b234f4756249b92850ed0"
server_key_iv: "58d917acb9dbbfd71e876ae2a8f3fc19"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.619220 9664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "f05b45999d234d77848e50f7c8f0e15e"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "47605130408b234f4756249b92850ed0"
server_key_iv: "58d917acb9dbbfd71e876ae2a8f3fc19"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.622373 9664 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:00.624349 9679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.625223 9664 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:00.625344 9664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "f05b45999d234d77848e50f7c8f0e15e"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "47605130408b234f4756249b92850ed0"
server_key_iv: "58d917acb9dbbfd71e876ae2a8f3fc19"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.625484 9664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:00.644454 9664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:00.647782 9664 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:00.647989 9664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:00.648589 9664 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:00.649633 9664 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:00.649722 9664 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.649806 9664 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:00.649858 9664 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.660475 9664 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:32913
I20251210 14:58:00.660534 9792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:32913 every 8 connection(s)
I20251210 14:58:00.661437 9664 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:00.663697 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9664
I20251210 14:58:00.663825 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:00.664116 6605 external_mini_cluster.cc:1468] Setting key 6d4a7b1a6aa109656d7c0eb1b8af24fa
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:00.675154 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:00.663279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:35547 (local address 127.6.115.126:35809)
1210 14:58:00.663440 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:00.663445 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:00.664155 (+ 710us) server_negotiation.cc:408] Connection header received
1210 14:58:00.665170 (+ 1015us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:00.665175 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:00.665230 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:00.665329 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:00.667769 (+ 2440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.668616 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:00.669542 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.669720 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:00.671840 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:00.671858 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:00.671860 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:00.671886 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:00.673312 (+ 1426us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.673804 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.673807 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.673808 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.673853 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.674150 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.674153 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.674155 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.674303 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:00.674405 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:00.674870 (+ 465us) server_negotiation.cc:300] Negotiation successful
1210 14:58:00.674999 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251210 14:58:00.675994 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:00.663568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35809 (local address 127.6.115.65:35547)
1210 14:58:00.664024 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:00.664064 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:00.664959 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:00.665641 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:00.665654 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:00.666211 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:00.667594 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:00.667615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.668778 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:00.668785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:00.669418 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:00.669430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.669637 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:00.670261 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:00.670284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:00.671695 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:00.673449 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:00.673454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:00.673467 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:00.673702 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:00.673935 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:00.673940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:00.673942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:00.674060 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:00.674425 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:00.674432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:00.674644 (+ 212us) client_negotiation.cc:769] Sending connection context
1210 14:58:00.674822 (+ 178us) client_negotiation.cc:240] Negotiation successful
1210 14:58:00.675245 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:58:00.677139 9793 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35809
I20251210 14:58:00.677433 9793 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:00.678035 9793 heartbeater.cc:507] Master 127.6.115.126:35809 requested a full tablet report, sending...
I20251210 14:58:00.679527 9600 ts_manager.cc:194] Registered new tserver with Master: f05b45999d234d77848e50f7c8f0e15e (127.6.115.65:32913)
I20251210 14:58:00.680701 9600 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:35547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:00.716351 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35809
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:00.816984 9800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:00.817185 9800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:00.817281 9800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:00.820518 9800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:00.820577 9800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:00.820699 9800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:00.824662 9800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:00.825615 9800 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:00.826344 9800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:00.832309 9806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:00.832365 9808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:00.832360 9805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:00.833025 9800 server_base.cc:1047] running on GCE node
I20251210 14:58:00.833434 9800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:00.833920 9800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:00.835068 9800 hybrid_clock.cc:648] HybridClock initialized: now 1765378680835057 us; error 25 us; skew 500 ppm
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:00.837654 9800 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:00.838601 9800 webserver.cc:492] Webserver started at http://127.6.115.66:39519/ using document root <none> and password file <none>
I20251210 14:58:00.839071 9800 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:00.839110 9800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:00.839241 9800 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:00.840734 9800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "d2028a8e9a1c46f3b77adf50c44c94f8"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "d2c11bd2b5716cc5a355011a0cd44ed8"
server_key_iv: "e89903685e00dc1e4019f1cd0451d955"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.841172 9800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "d2028a8e9a1c46f3b77adf50c44c94f8"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "d2c11bd2b5716cc5a355011a0cd44ed8"
server_key_iv: "e89903685e00dc1e4019f1cd0451d955"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.844265 9800 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:00.846280 9815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.847198 9800 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:00.847281 9800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "d2028a8e9a1c46f3b77adf50c44c94f8"
format_stamp: "Formatted at 2025-12-10 14:58:00 on dist-test-slave-nb51"
server_key: "d2c11bd2b5716cc5a355011a0cd44ed8"
server_key_iv: "e89903685e00dc1e4019f1cd0451d955"
server_key_version: "encryptionkey@0"
I20251210 14:58:00.847349 9800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:00.860427 9800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:00.863090 9800 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:00.863222 9800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:00.863662 9800 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:00.864391 9800 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:00.864434 9800 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.864470 9800 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:00.864526 9800 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:00.872745 9800 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:36977
I20251210 14:58:00.872769 9928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:36977 every 8 connection(s)
I20251210 14:58:00.873601 9800 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
Dec 10 14:58:00 dist-test-slave-nb51 krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:00.881487 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9800
I20251210 14:58:00.881583 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:00.881798 6605 external_mini_cluster.cc:1468] Setting key f8eb31f89f5b46ef897f2b3026fe64f2
I20251210 14:58:00.885612 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:00.875264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:60407 (local address 127.6.115.126:35809)
1210 14:58:00.875428 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:00.875433 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:00.876109 (+ 676us) server_negotiation.cc:408] Connection header received
1210 14:58:00.876904 (+ 795us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:00.876906 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:00.876955 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:00.877034 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:00.878622 (+ 1588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.879119 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:00.879729 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.879863 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:00.881912 (+ 2049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:00.881932 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:00.881935 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:00.881964 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:00.883700 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.884138 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.884144 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.884149 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.884203 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:00.884466 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:00.884469 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:00.884471 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:00.884656 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:00.884756 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:00.885390 (+ 634us) server_negotiation.cc:300] Negotiation successful
1210 14:58:00.885502 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251210 14:58:00.886523 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:00.875533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35809 (local address 127.6.115.66:60407)
1210 14:58:00.875957 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:00.875990 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:00.876669 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:00.877208 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:00.877215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:00.877623 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:00.878461 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:00.878474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.879228 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:00.879231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:00.879629 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:00.879636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:00.879794 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:00.880305 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:00.880323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:00.881755 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:00.883816 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:00.883821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:00.883830 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:00.884049 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:00.884293 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:00.884296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:00.884297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:00.884374 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:00.884763 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:00.884771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:00.885059 (+ 288us) client_negotiation.cc:769] Sending connection context
1210 14:58:00.885327 (+ 268us) client_negotiation.cc:240] Negotiation successful
1210 14:58:00.885633 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20251210 14:58:00.887614 9929 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35809
I20251210 14:58:00.887856 9929 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:00.888335 9929 heartbeater.cc:507] Master 127.6.115.126:35809 requested a full tablet report, sending...
I20251210 14:58:00.889274 9600 ts_manager.cc:194] Registered new tserver with Master: d2028a8e9a1c46f3b77adf50c44c94f8 (127.6.115.66:36977)
I20251210 14:58:00.889823 9600 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:60407
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:00.926571 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:35809
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:01.027843 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:01.028050 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:01.028138 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:01.031179 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:01.031237 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:01.031347 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:01.035311 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:35809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:01.036422 9936 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:01.037250 9936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:01.043465 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:01.043533 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:01.043496 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:01.043685 9936 server_base.cc:1047] running on GCE node
I20251210 14:58:01.044072 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:01.044584 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:01.045738 9936 hybrid_clock.cc:648] HybridClock initialized: now 1765378681045725 us; error 32 us; skew 500 ppm
Dec 10 14:58:01 dist-test-slave-nb51 krb5kdc[9552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:01.048414 9936 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:01.049391 9936 webserver.cc:492] Webserver started at http://127.6.115.67:42703/ using document root <none> and password file <none>
I20251210 14:58:01.049926 9936 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:01.049978 9936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:01.050132 9936 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:01.051901 9936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "b9028ceb21a84dfeafe31e87193f3524"
format_stamp: "Formatted at 2025-12-10 14:58:01 on dist-test-slave-nb51"
server_key: "5aa2d39f9dbdb66a7d8c77dbccfc5d8a"
server_key_iv: "4b1986d747e11dfe001d4cd6b2de4c6d"
server_key_version: "encryptionkey@0"
I20251210 14:58:01.052357 9936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "b9028ceb21a84dfeafe31e87193f3524"
format_stamp: "Formatted at 2025-12-10 14:58:01 on dist-test-slave-nb51"
server_key: "5aa2d39f9dbdb66a7d8c77dbccfc5d8a"
server_key_iv: "4b1986d747e11dfe001d4cd6b2de4c6d"
server_key_version: "encryptionkey@0"
I20251210 14:58:01.055732 9936 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:01.057860 9951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:01.058856 9936 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:01.058959 9936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "b9028ceb21a84dfeafe31e87193f3524"
format_stamp: "Formatted at 2025-12-10 14:58:01 on dist-test-slave-nb51"
server_key: "5aa2d39f9dbdb66a7d8c77dbccfc5d8a"
server_key_iv: "4b1986d747e11dfe001d4cd6b2de4c6d"
server_key_version: "encryptionkey@0"
I20251210 14:58:01.059072 9936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:01.075743 9936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:01.078472 9936 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:01.078655 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:01.079210 9936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:01.080106 9936 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:01.080174 9936 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:01.080221 9936 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:01.080269 9936 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:01.089593 9936 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:32805
I20251210 14:58:01.089620 10064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:32805 every 8 connection(s)
I20251210 14:58:01.090492 9936 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:01.091212 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 9936
I20251210 14:58:01.091333 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:01.091586 6605 external_mini_cluster.cc:1468] Setting key 7088f9b5b7979c4057a65df1e6d677a0
Dec 10 14:58:01 dist-test-slave-nb51 krb5kdc[9552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:01.102380 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:01.092329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:45325 (local address 127.6.115.126:35809)
1210 14:58:01.092491 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:01.092498 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:01.093287 (+ 789us) server_negotiation.cc:408] Connection header received
1210 14:58:01.094210 (+ 923us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:01.094213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:01.094275 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:01.094357 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:01.095883 (+ 1526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:01.096366 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:01.096993 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:01.097169 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:01.099211 (+ 2042us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:01.099224 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:01.099226 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:01.099247 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:01.100638 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:01.101048 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:01.101052 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:01.101053 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:01.101095 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:01.101416 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:01.101419 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:01.101421 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:01.101554 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:01.101652 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:01.102102 (+ 450us) server_negotiation.cc:300] Negotiation successful
1210 14:58:01.102218 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:58:01.102901 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:01.092670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:35809 (local address 127.6.115.67:45325)
1210 14:58:01.093168 (+ 498us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:01.093203 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:01.094026 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:01.094516 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:01.094524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:01.094866 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:01.095752 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:01.095764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:01.096480 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:01.096482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:01.096886 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:01.096893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:01.097092 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:01.097716 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:01.097738 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:01.099070 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:01.100744 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:01.100749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:01.100759 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:01.100958 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:01.101196 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:01.101200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:01.101202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:01.101299 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:01.101666 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:01.101673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:01.101878 (+ 205us) client_negotiation.cc:769] Sending connection context
1210 14:58:01.102058 (+ 180us) client_negotiation.cc:240] Negotiation successful
1210 14:58:01.102284 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"spinlock_wait_cycles":31488,"thread_start_us":136,"threads_started":1}
I20251210 14:58:01.103776 10065 heartbeater.cc:344] Connected to a master server at 127.6.115.126:35809
I20251210 14:58:01.103983 10065 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:01.104414 10065 heartbeater.cc:507] Master 127.6.115.126:35809 requested a full tablet report, sending...
I20251210 14:58:01.105352 9600 ts_manager.cc:194] Registered new tserver with Master: b9028ceb21a84dfeafe31e87193f3524 (127.6.115.67:32805)
I20251210 14:58:01.105882 9600 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:45325
I20251210 14:58:01.116600 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251210 14:58:01.242717 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:01.239494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.6.115.126:35809)
1210 14:58:01.239660 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:01.239665 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:01.240465 (+ 800us) server_negotiation.cc:408] Connection header received
1210 14:58:01.241238 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:01.241241 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:01.241288 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:01.241384 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:01.242599 (+ 1215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41684: BlockingRecv error: recv got EOF from 127.0.0.1:41684 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20251210 14:58:01.243942 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:01.243268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.6.115.126:35809)
1210 14:58:01.243372 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:01.243376 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:01.243388 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:01.243470 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:01.243472 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:01.243508 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:01.243586 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:01.243874 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41698: BlockingRecv error: recv got EOF from 127.0.0.1:41698 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20251210 14:58:01.244750 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:01.244199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.6.115.126:35809)
1210 14:58:01.244305 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:01.244308 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:01.244319 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:01.244345 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:01.244347 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:01.244382 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:01.244433 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:01.244677 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41704: BlockingRecv error: recv got EOF from 127.0.0.1:41704 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20251210 14:58:01.248207 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:01.247513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.6.115.126:35809)
1210 14:58:01.247638 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:01.247642 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:01.247656 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:58:01.247707 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:01.247710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:01.247741 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:01.247802 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:01.248152 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41712: BlockingRecv error: recv got EOF from 127.0.0.1:41712 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20251210 14:58:01.255767 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9664
I20251210 14:58:01.261037 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9800
I20251210 14:58:01.266403 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9936
I20251210 14:58:01.271350 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 9568
2025-12-10T14:58:01Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3142 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:01 dist-test-slave-nb51 krb5kdc[10088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:01 dist-test-slave-nb51 krb5kdc[10088](info): set up 2 sockets
Dec 10 14:58:01 dist-test-slave-nb51 krb5kdc[10088](info): commencing operation
krb5kdc: starting...
W20251210 14:58:03.314693 6605 mini_kdc.cc:121] Time spent starting KDC: real 2.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20251210 14:58:03.638764 6605 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.324s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20251210 14:58:03.779522 6605 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.141s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20251210 14:58:04.035310 6605 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.256s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/test-user.keytab.
W20251210 14:58:04.904325 6605 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.869s user 0.001s sys 0.001s
Dec 10 14:58:04 dist-test-slave-nb51 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab.
W20251210 14:58:05.477656 6605 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.6.115.126: real 0.553s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:05.499931 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43447
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:05.603879 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:05.604063 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:05.604115 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:05.607717 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:05.607793 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:05.607833 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:05.607872 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:05.607910 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:05.613957 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:43447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:05.614979 10104 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:05.615813 10104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:05.621377 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:05.621397 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:05.621434 10104 server_base.cc:1047] running on GCE node
W20251210 14:58:05.621351 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:05.622077 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:05.623006 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:05.624176 10104 hybrid_clock.cc:648] HybridClock initialized: now 1765378685624153 us; error 37 us; skew 500 ppm
Dec 10 14:58:05 dist-test-slave-nb51 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378685, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:05.626778 10104 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:05.627784 10104 webserver.cc:492] Webserver started at http://127.6.115.126:35123/ using document root <none> and password file <none>
I20251210 14:58:05.628358 10104 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:05.628414 10104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:05.628592 10104 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:05.630337 10104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "e33648b015e44433947ffcb0f2167064"
format_stamp: "Formatted at 2025-12-10 14:58:05 on dist-test-slave-nb51"
server_key: "d77090c275fcc47a87fda66a70913a70"
server_key_iv: "4807024e20be33aa107124ca362d6cc9"
server_key_version: "encryptionkey@0"
I20251210 14:58:05.630812 10104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "e33648b015e44433947ffcb0f2167064"
format_stamp: "Formatted at 2025-12-10 14:58:05 on dist-test-slave-nb51"
server_key: "d77090c275fcc47a87fda66a70913a70"
server_key_iv: "4807024e20be33aa107124ca362d6cc9"
server_key_version: "encryptionkey@0"
I20251210 14:58:05.634204 10104 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251210 14:58:05.636313 10119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:05.637327 10104 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:05.637454 10104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "e33648b015e44433947ffcb0f2167064"
format_stamp: "Formatted at 2025-12-10 14:58:05 on dist-test-slave-nb51"
server_key: "d77090c275fcc47a87fda66a70913a70"
server_key_iv: "4807024e20be33aa107124ca362d6cc9"
server_key_version: "encryptionkey@0"
I20251210 14:58:05.637534 10104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:05.656533 10104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:05.663136 10104 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:05.663319 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:05.670202 10171 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:43447 every 8 connection(s)
I20251210 14:58:05.670202 10104 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:43447
I20251210 14:58:05.671248 10104 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:05.674072 10172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:05.674813 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10104
I20251210 14:58:05.674937 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:05.675261 6605 external_mini_cluster.cc:1468] Setting key fd5abae85fd6ee50add78c405abb105a
I20251210 14:58:05.680685 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064: Bootstrap starting.
Dec 10 14:58:05 dist-test-slave-nb51 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:05.683351 10172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:05.684165 10172 log.cc:826] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:05.686060 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064: No bootstrap required, opened a new log
I20251210 14:58:05.687930 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:05.676476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.6.115.126:43447)
1210 14:58:05.676837 (+ 361us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:05.676846 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:05.676880 (+ 34us) server_negotiation.cc:408] Connection header received
1210 14:58:05.677449 (+ 569us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:05.677467 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:05.677734 (+ 267us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:05.677993 (+ 259us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:05.679143 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:05.679907 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:05.680757 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:05.681071 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:05.683681 (+ 2610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:05.683697 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:05.683720 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:05.683749 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:05.685475 (+ 1726us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:05.685957 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:05.685961 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:05.685965 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:05.686055 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:05.686323 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:05.686326 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:05.686328 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:05.686622 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:05.686766 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:05.687095 (+ 329us) server_negotiation.cc:300] Negotiation successful
1210 14:58:05.687332 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20251210 14:58:05.688665 10172 raft_consensus.cc:359] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } }
I20251210 14:58:05.688829 10172 raft_consensus.cc:385] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:05.688884 10172 raft_consensus.cc:740] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33648b015e44433947ffcb0f2167064, State: Initialized, Role: FOLLOWER
I20251210 14:58:05.689308 10172 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } }
I20251210 14:58:05.689529 10172 raft_consensus.cc:399] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:05.689607 10172 raft_consensus.cc:493] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:05.689714 10172 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:05.690459 6605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251210 14:58:05.690544 10172 raft_consensus.cc:515] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } }
I20251210 14:58:05.690910 10172 leader_election.cc:304] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e33648b015e44433947ffcb0f2167064; no voters:
I20251210 14:58:05.691238 10172 leader_election.cc:290] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:05.691325 10177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:05.691524 10177 raft_consensus.cc:697] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [term 1 LEADER]: Becoming Leader. State: Replica: e33648b015e44433947ffcb0f2167064, State: Running, Role: LEADER
I20251210 14:58:05.691828 10177 consensus_queue.cc:237] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } }
I20251210 14:58:05.692483 10172 sys_catalog.cc:565] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:05.693513 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e33648b015e44433947ffcb0f2167064. Latest consensus state: current_term: 1 leader_uuid: "e33648b015e44433947ffcb0f2167064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } } }
I20251210 14:58:05.693621 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:05.693819 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e33648b015e44433947ffcb0f2167064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33648b015e44433947ffcb0f2167064" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43447 } } }
I20251210 14:58:05.693919 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:05.694388 10187 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:05.698186 10187 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:05.703766 10187 catalog_manager.cc:1357] Generated new cluster ID: 39d232b5040445738575a65337fbc166
I20251210 14:58:05.703876 10187 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:05.710554 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:05.706367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.6.115.126:43447)
1210 14:58:05.706464 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:05.706467 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:05.706763 (+ 296us) server_negotiation.cc:408] Connection header received
1210 14:58:05.706872 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:05.706876 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:05.706927 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:05.707000 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:05.707621 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:05.708081 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:05.708689 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:05.708935 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:05.709525 (+ 590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:05.709539 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:05.709541 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:05.709929 (+ 388us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:05.709956 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:05.709970 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:05.710231 (+ 261us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:05.710342 (+ 111us) server_negotiation.cc:300] Negotiation successful
1210 14:58:05.710415 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251210 14:58:05.725484 10187 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:05.726375 10187 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:05.731252 10187 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e33648b015e44433947ffcb0f2167064: Generated new TSK 0
I20251210 14:58:05.731891 10187 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:58:05.740938 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10104
2025-12-10T14:58:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (4467 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:05 dist-test-slave-nb51 krb5kdc[10205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:05 dist-test-slave-nb51 krb5kdc[10205](info): set up 2 sockets
Dec 10 14:58:05 dist-test-slave-nb51 krb5kdc[10205](info): commencing operation
krb5kdc: starting...
W20251210 14:58:07.701678 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:07 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:07.824549 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39375
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:39375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:07.926196 10221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:07.926446 10221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:07.926535 10221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:07.929651 10221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:07.929710 10221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:07.929759 10221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:07.929808 10221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:07.929848 10221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:07.934033 10221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:39375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:07.935016 10221 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:07.935803 10221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:07.941083 10229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:07.941069 10227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:07.941072 10226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:07.941139 10221 server_base.cc:1047] running on GCE node
I20251210 14:58:07.941859 10221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:07.942685 10221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:07.943848 10221 hybrid_clock.cc:648] HybridClock initialized: now 1765378687943828 us; error 35 us; skew 500 ppm
Dec 10 14:58:07 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378687, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:07.946362 10221 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:07.947340 10221 webserver.cc:492] Webserver started at http://127.6.115.126:40301/ using document root <none> and password file <none>
I20251210 14:58:07.947836 10221 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:07.947889 10221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:07.948052 10221 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:07.949647 10221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02"
format_stamp: "Formatted at 2025-12-10 14:58:07 on dist-test-slave-nb51"
server_key: "012b527b5275f62280eb1db880c5e178"
server_key_iv: "cc92ff6b11bb6c56045876b6501c8d9c"
server_key_version: "encryptionkey@0"
I20251210 14:58:07.950107 10221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02"
format_stamp: "Formatted at 2025-12-10 14:58:07 on dist-test-slave-nb51"
server_key: "012b527b5275f62280eb1db880c5e178"
server_key_iv: "cc92ff6b11bb6c56045876b6501c8d9c"
server_key_version: "encryptionkey@0"
I20251210 14:58:07.953284 10221 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:58:07.955399 10236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:07.956284 10221 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:07.956396 10221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02"
format_stamp: "Formatted at 2025-12-10 14:58:07 on dist-test-slave-nb51"
server_key: "012b527b5275f62280eb1db880c5e178"
server_key_iv: "cc92ff6b11bb6c56045876b6501c8d9c"
server_key_version: "encryptionkey@0"
I20251210 14:58:07.956480 10221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:07.978459 10221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:07.981361 10221 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:07.981591 10221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:07.988426 10221 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:39375
I20251210 14:58:07.988430 10288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:39375 every 8 connection(s)
I20251210 14:58:07.989473 10221 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:07.992125 10289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:07.997318 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02: Bootstrap starting.
I20251210 14:58:07.999118 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10221
I20251210 14:58:07.999257 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:07.999500 6605 external_mini_cluster.cc:1468] Setting key 2b017851785fdc08aac13792aaefcb52
I20251210 14:58:07.999819 10289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:08.000659 10289 log.cc:826] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:08.002343 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02: No bootstrap required, opened a new log
I20251210 14:58:08.004594 10289 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } }
I20251210 14:58:08.004791 10289 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:08.004894 10289 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf09f8ebbb34d28ab473d8ae41a2b02, State: Initialized, Role: FOLLOWER
I20251210 14:58:08.005295 10289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } }
I20251210 14:58:08.005460 10289 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:08.005551 10289 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:08.005642 10289 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:08.006398 10289 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } }
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:08.006757 10289 leader_election.cc:304] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bf09f8ebbb34d28ab473d8ae41a2b02; no voters:
I20251210 14:58:08.007117 10289 leader_election.cc:290] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:08.007208 10294 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:08.007404 10294 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf09f8ebbb34d28ab473d8ae41a2b02, State: Running, Role: LEADER
I20251210 14:58:08.007716 10294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } }
I20251210 14:58:08.008314 10289 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:08.009584 10296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } } }
I20251210 14:58:08.009702 10296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:08.010011 10295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf09f8ebbb34d28ab473d8ae41a2b02. Latest consensus state: current_term: 1 leader_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf09f8ebbb34d28ab473d8ae41a2b02" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39375 } } }
I20251210 14:58:08.010080 10303 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:08.013188 10303 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:08.013249 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.000534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.6.115.126:39375)
1210 14:58:08.000948 (+ 414us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:08.000957 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:08.000988 (+ 31us) server_negotiation.cc:408] Connection header received
1210 14:58:08.001555 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:08.001575 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:08.001852 (+ 277us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:08.002139 (+ 287us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:08.003103 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.003848 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.004474 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.004747 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.007399 (+ 2652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:08.007423 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:08.007438 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:08.007478 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:08.010295 (+ 2817us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.010870 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.010876 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.010881 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.010967 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.011278 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.011281 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.011283 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.011708 (+ 425us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:08.011894 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:08.012293 (+ 399us) server_negotiation.cc:300] Negotiation successful
1210 14:58:08.012553 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20251210 14:58:08.010176 10295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:08.018113 10303 catalog_manager.cc:1357] Generated new cluster ID: 029fd6e92e4a40c1b13dc5d9ccf90ebc
I20251210 14:58:08.018173 10303 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:08.029394 10303 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:08.030174 10303 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:08.047470 10303 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0bf09f8ebbb34d28ab473d8ae41a2b02: Generated new TSK 0
I20251210 14:58:08.048137 10303 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:08.098263 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39375
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:08.200313 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:08.200520 10317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:08.200577 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:08.203917 10317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:08.203977 10317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:08.204069 10317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:08.208089 10317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:08.209046 10317 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:08.209909 10317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:08.215961 10322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.215961 10323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.215975 10325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:08.216257 10317 server_base.cc:1047] running on GCE node
I20251210 14:58:08.216656 10317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:08.217170 10317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:08.218327 10317 hybrid_clock.cc:648] HybridClock initialized: now 1765378688218307 us; error 45 us; skew 500 ppm
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:08.220837 10317 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:08.221863 10317 webserver.cc:492] Webserver started at http://127.6.115.65:45735/ using document root <none> and password file <none>
I20251210 14:58:08.222373 10317 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:08.222443 10317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:08.222616 10317 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:08.224330 10317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "0d08912fbe3a404f96a47dfc7efab9ba"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "b00420d25f63ac7509645618cf80820e"
server_key_iv: "7654331f1c743e05ffd9812b3ce4b1f1"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.224785 10317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "0d08912fbe3a404f96a47dfc7efab9ba"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "b00420d25f63ac7509645618cf80820e"
server_key_iv: "7654331f1c743e05ffd9812b3ce4b1f1"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.227869 10317 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:08.229846 10332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.230756 10317 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:08.230875 10317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "0d08912fbe3a404f96a47dfc7efab9ba"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "b00420d25f63ac7509645618cf80820e"
server_key_iv: "7654331f1c743e05ffd9812b3ce4b1f1"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.230962 10317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:08.244781 10317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:08.247571 10317 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:08.247745 10317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:08.248270 10317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:08.249101 10317 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:08.249150 10317 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.249217 10317 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:08.249243 10317 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.258070 10317 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:37785
I20251210 14:58:08.258101 10445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:37785 every 8 connection(s)
I20251210 14:58:08.258950 10317 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:08.263612 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10317
I20251210 14:58:08.263760 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:08.264070 6605 external_mini_cluster.cc:1468] Setting key 9a2e0af87549865f234e7c32e5aaa824
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:08.272066 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.260591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:37717 (local address 127.6.115.126:39375)
1210 14:58:08.260749 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:08.260752 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:08.261425 (+ 673us) server_negotiation.cc:408] Connection header received
1210 14:58:08.262285 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:08.262290 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:08.262356 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:08.262457 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:08.264353 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.264970 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.265828 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.265960 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.268538 (+ 2578us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:08.268553 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:08.268556 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:08.268580 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:08.270177 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.270740 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.270744 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.270747 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.270795 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.271086 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.271089 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.271091 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.271237 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:08.271329 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:08.271776 (+ 447us) server_negotiation.cc:300] Negotiation successful
1210 14:58:08.271910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:08.272665 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.260838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39375 (local address 127.6.115.65:37717)
1210 14:58:08.261262 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:08.261297 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:08.262081 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:08.262626 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:08.262634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:08.263003 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:08.264197 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.264216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.265140 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.265143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:08.265693 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.265704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.265913 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.266505 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:08.266539 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:08.268396 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:08.270367 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.270381 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.270391 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.270615 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.270881 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.270887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.270889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.270981 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.271338 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:08.271346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:08.271550 (+ 204us) client_negotiation.cc:769] Sending connection context
1210 14:58:08.271727 (+ 177us) client_negotiation.cc:240] Negotiation successful
1210 14:58:08.271969 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20251210 14:58:08.273725 10446 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39375
I20251210 14:58:08.273936 10446 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:08.274397 10446 heartbeater.cc:507] Master 127.6.115.126:39375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:58:08.275846 10253 ts_manager.cc:194] Registered new tserver with Master: 0d08912fbe3a404f96a47dfc7efab9ba (127.6.115.65:37785)
I20251210 14:58:08.277347 10253 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:37717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:08.308679 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39375
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:08.412178 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:08.412386 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:08.412443 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:08.415644 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:08.415699 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:08.415803 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:08.419801 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:08.420744 10453 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:08.421542 10453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:08.427613 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.427589 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.427589 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:08.427651 10453 server_base.cc:1047] running on GCE node
I20251210 14:58:08.428086 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:08.428561 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:08.429706 10453 hybrid_clock.cc:648] HybridClock initialized: now 1765378688429693 us; error 29 us; skew 500 ppm
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:08.432334 10453 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:08.433331 10453 webserver.cc:492] Webserver started at http://127.6.115.66:37805/ using document root <none> and password file <none>
I20251210 14:58:08.433863 10453 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:08.433923 10453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:08.434119 10453 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:08.435842 10453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "28df98276c32489796b18ef001af8ca7"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "088abe3a38d76530b9e28cf333f8b742"
server_key_iv: "c7a4b18b2aff9303a18a4570c5c5a44d"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.436272 10453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "28df98276c32489796b18ef001af8ca7"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "088abe3a38d76530b9e28cf333f8b742"
server_key_iv: "c7a4b18b2aff9303a18a4570c5c5a44d"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.439400 10453 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:08.441550 10468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.442543 10453 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:08.442641 10453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "28df98276c32489796b18ef001af8ca7"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "088abe3a38d76530b9e28cf333f8b742"
server_key_iv: "c7a4b18b2aff9303a18a4570c5c5a44d"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.442734 10453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:08.474212 10453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:08.477265 10453 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:08.477481 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:08.478029 10453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:08.478878 10453 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:08.478922 10453 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.478965 10453 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:08.478999 10453 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.487946 10453 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:37085
I20251210 14:58:08.488129 10581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:37085 every 8 connection(s)
I20251210 14:58:08.489219 10453 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:08.494014 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10453
I20251210 14:58:08.494158 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:08.494380 6605 external_mini_cluster.cc:1468] Setting key 22a0941012fd4f1a93c8a6d919d29d68
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:08.501468 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.491100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:54199 (local address 127.6.115.126:39375)
1210 14:58:08.491230 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:08.491234 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:08.491832 (+ 598us) server_negotiation.cc:408] Connection header received
1210 14:58:08.492668 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:08.492671 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:08.492733 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:08.492837 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:08.494539 (+ 1702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.495136 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.495834 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.496006 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.498082 (+ 2076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:08.498098 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:08.498100 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:08.498123 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:08.499701 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.500183 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.500187 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.500188 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.500229 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.500500 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.500503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.500505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.500641 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:08.500729 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:08.501166 (+ 437us) server_negotiation.cc:300] Negotiation successful
1210 14:58:08.501292 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:08.502068 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.491258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39375 (local address 127.6.115.66:54199)
1210 14:58:08.491679 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:08.491714 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:08.492471 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:08.493000 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:08.493008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:08.493429 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:08.494407 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.494421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.495259 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.495262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:08.495719 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.495727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.495929 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.496533 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:08.496555 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:08.497936 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:08.499822 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.499827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.499838 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.500087 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.500312 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.500316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.500318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.500416 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.500738 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:08.500747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:08.500943 (+ 196us) client_negotiation.cc:769] Sending connection context
1210 14:58:08.501121 (+ 178us) client_negotiation.cc:240] Negotiation successful
1210 14:58:08.501412 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:58:08.503109 10582 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39375
I20251210 14:58:08.503350 10582 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:08.503837 10582 heartbeater.cc:507] Master 127.6.115.126:39375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
I20251210 14:58:08.504824 10253 ts_manager.cc:194] Registered new tserver with Master: 28df98276c32489796b18ef001af8ca7 (127.6.115.66:37085)
I20251210 14:58:08.505486 10253 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:54199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:08.537753 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39375
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:08.636245 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:08.636451 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:08.636518 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:08.639698 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:08.639750 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:08.639861 10589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:08.643720 10589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:08.644645 10589 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:08.645434 10589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:08.651541 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.651530 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:08.651526 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:08.651746 10589 server_base.cc:1047] running on GCE node
I20251210 14:58:08.652149 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:08.652642 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:08.653813 10589 hybrid_clock.cc:648] HybridClock initialized: now 1765378688653804 us; error 28 us; skew 500 ppm
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:08.656486 10589 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:08.657501 10589 webserver.cc:492] Webserver started at http://127.6.115.67:32797/ using document root <none> and password file <none>
I20251210 14:58:08.658007 10589 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:08.658056 10589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:08.658247 10589 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:08.659821 10589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "d9d5ef1a35ea43a5971cb3e80560f722"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "0018cca3b84046ff6f42383a8477c5db"
server_key_iv: "e98fd6aa1761e92fb21a4b98c0395b8e"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.660270 10589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d5ef1a35ea43a5971cb3e80560f722"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "0018cca3b84046ff6f42383a8477c5db"
server_key_iv: "e98fd6aa1761e92fb21a4b98c0395b8e"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.663646 10589 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251210 14:58:08.665752 10604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.666666 10589 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251210 14:58:08.666780 10589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "d9d5ef1a35ea43a5971cb3e80560f722"
format_stamp: "Formatted at 2025-12-10 14:58:08 on dist-test-slave-nb51"
server_key: "0018cca3b84046ff6f42383a8477c5db"
server_key_iv: "e98fd6aa1761e92fb21a4b98c0395b8e"
server_key_version: "encryptionkey@0"
I20251210 14:58:08.666863 10589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:08.682448 10589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:08.685196 10589 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:08.685395 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:08.685869 10589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:08.686731 10589 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:08.686781 10589 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.686841 10589 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:08.686869 10589 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:08.695353 10589 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:37351
I20251210 14:58:08.695376 10717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:37351 every 8 connection(s)
I20251210 14:58:08.696202 10589 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:08.702416 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10589
I20251210 14:58:08.702548 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:08.702780 6605 external_mini_cluster.cc:1468] Setting key 2a32e689926a6cd545681210ae5deff1
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:08.707568 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.697846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:59551 (local address 127.6.115.126:39375)
1210 14:58:08.698012 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:08.698017 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:08.698608 (+ 591us) server_negotiation.cc:408] Connection header received
1210 14:58:08.699345 (+ 737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:08.699347 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:08.699396 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:08.699470 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:08.701148 (+ 1678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.701683 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.702347 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.702546 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.704626 (+ 2080us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:08.704642 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:08.704659 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:08.704703 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:08.706151 (+ 1448us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.706577 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.706580 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.706581 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.706615 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.706825 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.706827 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.706828 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.706944 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:08.707027 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:08.707380 (+ 353us) server_negotiation.cc:300] Negotiation successful
1210 14:58:08.707464 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251210 14:58:08.708196 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.698084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39375 (local address 127.6.115.67:59551)
1210 14:58:08.698475 (+ 391us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:08.698507 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:08.699171 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:08.699671 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:08.699679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:08.700044 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:08.700990 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.701010 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.701805 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.701808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:08.702231 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:08.702238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.702396 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.703038 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:08.703059 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:08.704506 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:08.706270 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.706274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.706282 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.706502 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.706675 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:08.706677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:08.706678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:08.706769 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:08.707031 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:08.707035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:08.707209 (+ 174us) client_negotiation.cc:769] Sending connection context
1210 14:58:08.707353 (+ 144us) client_negotiation.cc:240] Negotiation successful
1210 14:58:08.707524 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20251210 14:58:08.709053 10718 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39375
I20251210 14:58:08.709241 10718 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:08.709707 10718 heartbeater.cc:507] Master 127.6.115.126:39375 requested a full tablet report, sending...
I20251210 14:58:08.710600 10253 ts_manager.cc:194] Registered new tserver with Master: d9d5ef1a35ea43a5971cb3e80560f722 (127.6.115.67:37351)
I20251210 14:58:08.711136 10253 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:59551
I20251210 14:58:08.716187 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378688, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:08.738531 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:08.731070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.6.115.126:39375)
1210 14:58:08.731227 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:08.731231 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:08.731340 (+ 109us) server_negotiation.cc:408] Connection header received
1210 14:58:08.731468 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:08.731471 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:08.731513 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:08.731582 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:08.732366 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.732850 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:08.733468 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:08.733673 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:08.735656 (+ 1983us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:08.735670 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:08.735672 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:08.735713 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:08.737197 (+ 1484us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.737577 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.737579 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.737581 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.737616 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:08.737799 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:08.737801 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:08.737802 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:08.737954 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:08.738052 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:08.738294 (+ 242us) server_negotiation.cc:300] Negotiation successful
1210 14:58:08.738379 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20251210 14:58:08.738919 10253 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42720
I20251210 14:58:08.739702 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10317
I20251210 14:58:08.745124 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10453
I20251210 14:58:08.750504 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10589
I20251210 14:58:08.755450 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10221
2025-12-10T14:58:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3016 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10730](info): set up 2 sockets
Dec 10 14:58:08 dist-test-slave-nb51 krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20251210 14:58:10.729667 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:10 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:10.856498 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:33513
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:33513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:10.956167 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:10.956363 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:10.956414 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:10.959571 10746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:10.959619 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:10.959642 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:10.959661 10746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:10.959676 10746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:10.963709 10746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:33513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:33513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:10.964677 10746 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:10.965474 10746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:10.970664 10751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:10.970741 10746 server_base.cc:1047] running on GCE node
W20251210 14:58:10.970650 10752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:10.970649 10754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:10.971364 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:10.972190 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:10.973332 10746 hybrid_clock.cc:648] HybridClock initialized: now 1765378690973320 us; error 30 us; skew 500 ppm
Dec 10 14:58:10 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378690, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:10.975725 10746 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:10.976657 10746 webserver.cc:492] Webserver started at http://127.6.115.126:43453/ using document root <none> and password file <none>
I20251210 14:58:10.977128 10746 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:10.977170 10746 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:10.977337 10746 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:10.978905 10746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "5ca0d63022fc41b0bf20146574a950c7"
format_stamp: "Formatted at 2025-12-10 14:58:10 on dist-test-slave-nb51"
server_key: "bbf7736cfb4f3989de5680f3abb1f787"
server_key_iv: "d882c872a500857d4d2c3493ac55cffd"
server_key_version: "encryptionkey@0"
I20251210 14:58:10.979408 10746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "5ca0d63022fc41b0bf20146574a950c7"
format_stamp: "Formatted at 2025-12-10 14:58:10 on dist-test-slave-nb51"
server_key: "bbf7736cfb4f3989de5680f3abb1f787"
server_key_iv: "d882c872a500857d4d2c3493ac55cffd"
server_key_version: "encryptionkey@0"
I20251210 14:58:10.982547 10746 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:10.984535 10761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:10.985558 10746 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:10.985663 10746 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "5ca0d63022fc41b0bf20146574a950c7"
format_stamp: "Formatted at 2025-12-10 14:58:10 on dist-test-slave-nb51"
server_key: "bbf7736cfb4f3989de5680f3abb1f787"
server_key_iv: "d882c872a500857d4d2c3493ac55cffd"
server_key_version: "encryptionkey@0"
I20251210 14:58:10.985751 10746 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:11.003862 10746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:11.010352 10746 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:11.010548 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:11.017516 10746 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:33513
I20251210 14:58:11.017530 10813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:33513 every 8 connection(s)
I20251210 14:58:11.018568 10746 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:11.021104 10814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:11.021253 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10746
I20251210 14:58:11.021399 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:11.021641 6605 external_mini_cluster.cc:1468] Setting key 91dd5946d16513a3f47caad9819bddad
I20251210 14:58:11.027462 10814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7: Bootstrap starting.
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:11.029803 10814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:11.030593 10814 log.cc:826] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:11.032260 10814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7: No bootstrap required, opened a new log
I20251210 14:58:11.033980 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.022837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.6.115.126:33513)
1210 14:58:11.023213 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.023222 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.023257 (+ 35us) server_negotiation.cc:408] Connection header received
1210 14:58:11.024008 (+ 751us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.024036 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.024397 (+ 361us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.024671 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.025654 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.026398 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.027257 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.027571 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.029840 (+ 2269us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.029868 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.029878 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.029935 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.031769 (+ 1834us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.032162 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.032166 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.032170 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.032234 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.032459 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.032462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.032464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.032752 (+ 288us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.032939 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.033181 (+ 242us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.033450 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20251210 14:58:11.034631 10814 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } }
I20251210 14:58:11.034824 10814 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:11.034881 10814 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca0d63022fc41b0bf20146574a950c7, State: Initialized, Role: FOLLOWER
I20251210 14:58:11.035388 10814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } }
I20251210 14:58:11.035533 10814 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:11.035609 10814 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:11.035709 10814 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:11.036834 10814 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } }
I20251210 14:58:11.037210 10814 leader_election.cc:304] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca0d63022fc41b0bf20146574a950c7; no voters:
I20251210 14:58:11.037477 10814 leader_election.cc:290] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:11.037568 10819 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:11.037760 10819 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca0d63022fc41b0bf20146574a950c7, State: Running, Role: LEADER
I20251210 14:58:11.038102 10819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } }
I20251210 14:58:11.038445 10814 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:11.039688 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca0d63022fc41b0bf20146574a950c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } } }
I20251210 14:58:11.039819 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:11.039781 10821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca0d63022fc41b0bf20146574a950c7. Latest consensus state: current_term: 1 leader_uuid: "5ca0d63022fc41b0bf20146574a950c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca0d63022fc41b0bf20146574a950c7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33513 } } }
I20251210 14:58:11.039870 10821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:11.040190 10828 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:11.042748 10828 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:11.047693 10828 catalog_manager.cc:1357] Generated new cluster ID: 084161526f5f4cc8b83a2bb99ea9789a
I20251210 14:58:11.047745 10828 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:11.065196 10828 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:11.066040 10828 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:11.074420 10828 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5ca0d63022fc41b0bf20146574a950c7: Generated new TSK 0
I20251210 14:58:11.075045 10828 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:11.120311 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33513
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:11.222146 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:11.222326 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:11.222397 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:11.225553 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:11.225612 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:11.225726 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:11.229691 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:33513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:11.230634 10842 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:11.231426 10842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:11.237479 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:11.237485 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:11.237543 10842 server_base.cc:1047] running on GCE node
W20251210 14:58:11.237497 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:11.238029 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:11.238554 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:11.239701 10842 hybrid_clock.cc:648] HybridClock initialized: now 1765378691239678 us; error 36 us; skew 500 ppm
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:11.242337 10842 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:11.243271 10842 webserver.cc:492] Webserver started at http://127.6.115.65:40185/ using document root <none> and password file <none>
I20251210 14:58:11.243747 10842 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:11.243813 10842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:11.243963 10842 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:11.245635 10842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "79ae7746da7745118880daaf10d8fcfa"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "93d44f782f4c72a760568fee87698e16"
server_key_iv: "5c12fcaedd91649d79b340ad4d619205"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.246086 10842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "79ae7746da7745118880daaf10d8fcfa"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "93d44f782f4c72a760568fee87698e16"
server_key_iv: "5c12fcaedd91649d79b340ad4d619205"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.249182 10842 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:58:11.251273 10857 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.252187 10842 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251210 14:58:11.252272 10842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "79ae7746da7745118880daaf10d8fcfa"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "93d44f782f4c72a760568fee87698e16"
server_key_iv: "5c12fcaedd91649d79b340ad4d619205"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.252343 10842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:11.267688 10842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:11.270251 10842 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:11.270396 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:11.270849 10842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:11.271663 10842 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:11.271714 10842 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.271754 10842 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:11.271790 10842 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.281031 10842 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:35213
I20251210 14:58:11.281057 10970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:35213 every 8 connection(s)
I20251210 14:58:11.282025 10842 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:11.285240 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10842
I20251210 14:58:11.285414 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:11.285657 6605 external_mini_cluster.cc:1468] Setting key b9fe65520566588d4a7ca5c4ad43a43c
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:11.294902 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.283646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:50793 (local address 127.6.115.126:33513)
1210 14:58:11.283789 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.283795 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.284412 (+ 617us) server_negotiation.cc:408] Connection header received
1210 14:58:11.285160 (+ 748us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.285165 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.285231 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.285339 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.286999 (+ 1660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.287797 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.288492 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.288732 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.290752 (+ 2020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.290777 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.290782 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.290817 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.292811 (+ 1994us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.293332 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.293336 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.293338 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.293438 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.293712 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.293716 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.293720 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.293953 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.294076 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.294531 (+ 455us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.294680 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:11.295265 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.283880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33513 (local address 127.6.115.65:50793)
1210 14:58:11.284287 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:11.284318 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:11.284961 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:11.285520 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:11.285528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:11.285878 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:11.286826 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.286845 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.287957 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.287960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:11.288367 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.288375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.288553 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.289104 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:11.289127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:11.290575 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:11.292994 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.292999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.293007 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.293226 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.293534 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.293537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.293538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.293622 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.294087 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:11.294092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:11.294286 (+ 194us) client_negotiation.cc:769] Sending connection context
1210 14:58:11.294447 (+ 161us) client_negotiation.cc:240] Negotiation successful
1210 14:58:11.294634 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:11.296346 10971 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33513
I20251210 14:58:11.296589 10971 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:11.297077 10971 heartbeater.cc:507] Master 127.6.115.126:33513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:58:11.298583 10778 ts_manager.cc:194] Registered new tserver with Master: 79ae7746da7745118880daaf10d8fcfa (127.6.115.65:35213)
I20251210 14:58:11.300011 10778 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:50793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:11.330447 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33513
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:11.429607 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:11.429800 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:11.429860 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:11.432929 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:11.432979 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:11.433050 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:11.436901 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:33513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:11.437778 10978 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:11.438493 10978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:11.444273 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:11.444273 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:11.444283 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:11.444578 10978 server_base.cc:1047] running on GCE node
I20251210 14:58:11.444921 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:11.445443 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:11.446595 10978 hybrid_clock.cc:648] HybridClock initialized: now 1765378691446572 us; error 42 us; skew 500 ppm
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:11.448989 10978 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:11.449971 10978 webserver.cc:492] Webserver started at http://127.6.115.66:43699/ using document root <none> and password file <none>
I20251210 14:58:11.450459 10978 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:11.450510 10978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:11.450675 10978 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:11.452196 10978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "53aa0c1a9c804393a3aa2ed5cff7c9a8"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "d723cdfbbfc96988001c88356d9244b8"
server_key_iv: "a028074272ada179d32c1b7218222a13"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.452623 10978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "53aa0c1a9c804393a3aa2ed5cff7c9a8"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "d723cdfbbfc96988001c88356d9244b8"
server_key_iv: "a028074272ada179d32c1b7218222a13"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.455711 10978 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251210 14:58:11.459508 10993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.460788 10978 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251210 14:58:11.460891 10978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "53aa0c1a9c804393a3aa2ed5cff7c9a8"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "d723cdfbbfc96988001c88356d9244b8"
server_key_iv: "a028074272ada179d32c1b7218222a13"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.461040 10978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:11.497023 10978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:11.499634 10978 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:11.499805 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:11.500317 10978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:11.501160 10978 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:11.501209 10978 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.501267 10978 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:11.501294 10978 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.509714 10978 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:37051
I20251210 14:58:11.509763 11106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:37051 every 8 connection(s)
I20251210 14:58:11.510599 10978 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:11.515368 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 10978
I20251210 14:58:11.515491 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:11.515756 6605 external_mini_cluster.cc:1468] Setting key fd09e7d195e343a22a36a21f47b86e92
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:11.523475 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.512176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:38653 (local address 127.6.115.126:33513)
1210 14:58:11.512333 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.512338 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.512960 (+ 622us) server_negotiation.cc:408] Connection header received
1210 14:58:11.513754 (+ 794us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.513758 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.513817 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.513895 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.515689 (+ 1794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.516319 (+ 630us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.517033 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.517195 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.519759 (+ 2564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.519780 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.519787 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.519845 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.521530 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.522146 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.522149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.522150 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.522190 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.522493 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.522495 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.522496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.522613 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.522674 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.523275 (+ 601us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.523364 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251210 14:58:11.524361 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.512414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33513 (local address 127.6.115.66:38653)
1210 14:58:11.512805 (+ 391us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:11.512858 (+ 53us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:11.513567 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:11.514050 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:11.514058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:11.514424 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:11.515536 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.515555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.516438 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.516441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:11.516922 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.516933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.517124 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.517724 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:11.517760 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:11.519615 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:11.521645 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.521654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.521666 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.522023 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.522283 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.522289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.522293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.522410 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.522725 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:11.522733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:11.523007 (+ 274us) client_negotiation.cc:769] Sending connection context
1210 14:58:11.523251 (+ 244us) client_negotiation.cc:240] Negotiation successful
1210 14:58:11.523537 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:58:11.525548 11107 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33513
I20251210 14:58:11.525779 11107 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:11.526248 11107 heartbeater.cc:507] Master 127.6.115.126:33513 requested a full tablet report, sending...
I20251210 14:58:11.527145 10778 ts_manager.cc:194] Registered new tserver with Master: 53aa0c1a9c804393a3aa2ed5cff7c9a8 (127.6.115.66:37051)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
I20251210 14:58:11.527700 10778 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:38653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:11.559875 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33513
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:11.660311 11114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:11.660480 11114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:11.660533 11114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:11.663640 11114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:11.663712 11114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:11.663815 11114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:11.667835 11114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:33513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:11.668735 11114 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:11.669528 11114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:11.677155 11120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:11.677170 11119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:11.677284 11122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:11.677814 11114 server_base.cc:1047] running on GCE node
I20251210 14:58:11.678241 11114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:11.678730 11114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:11.679863 11114 hybrid_clock.cc:648] HybridClock initialized: now 1765378691679849 us; error 28 us; skew 500 ppm
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:11.682861 11114 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:11.684324 11114 webserver.cc:492] Webserver started at http://127.6.115.67:40875/ using document root <none> and password file <none>
I20251210 14:58:11.685139 11114 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:11.685197 11114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:11.685411 11114 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:11.687371 11114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "253a7a34ab9e4df188e60264258cc0cf"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "a3d551ae1ef4aba60f80aa4e3667e763"
server_key_iv: "57e7c0a7717cdfa17e7a736ffc1832dc"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.687935 11114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "253a7a34ab9e4df188e60264258cc0cf"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "a3d551ae1ef4aba60f80aa4e3667e763"
server_key_iv: "57e7c0a7717cdfa17e7a736ffc1832dc"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.692332 11114 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20251210 14:58:11.694795 11129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.696014 11114 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:11.696153 11114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "253a7a34ab9e4df188e60264258cc0cf"
format_stamp: "Formatted at 2025-12-10 14:58:11 on dist-test-slave-nb51"
server_key: "a3d551ae1ef4aba60f80aa4e3667e763"
server_key_iv: "57e7c0a7717cdfa17e7a736ffc1832dc"
server_key_version: "encryptionkey@0"
I20251210 14:58:11.696254 11114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:11.717809 11114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:11.720274 11114 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:11.720433 11114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:11.720924 11114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:11.721817 11114 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:11.721863 11114 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.721906 11114 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:11.721940 11114 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:11.730321 11114 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:43901
I20251210 14:58:11.730363 11242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:43901 every 8 connection(s)
I20251210 14:58:11.731225 11114 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:11.734723 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11114
I20251210 14:58:11.734839 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:11.735059 6605 external_mini_cluster.cc:1468] Setting key 89ff7b8434de818c25aa80641c4dcd49
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:11.743010 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.732754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:36615 (local address 127.6.115.126:33513)
1210 14:58:11.732903 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.732907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.733570 (+ 663us) server_negotiation.cc:408] Connection header received
1210 14:58:11.734397 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.734400 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.734456 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.734549 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.736361 (+ 1812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.736911 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.737577 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.737758 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.739847 (+ 2089us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.739860 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.739862 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.739885 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.741309 (+ 1424us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.741788 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.741792 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.741793 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.741833 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.742079 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.742082 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.742084 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.742214 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.742302 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.742737 (+ 435us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.742846 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:11.743512 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.732993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33513 (local address 127.6.115.67:36615)
1210 14:58:11.733451 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:11.733483 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:11.734201 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:11.734742 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:11.734749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:11.735189 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:11.736223 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.736240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.737024 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.737028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:11.737477 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:11.737485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.737677 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.738314 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:11.738335 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:11.739691 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:11.741443 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.741459 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.741468 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.741687 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.741910 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:11.741915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:11.741917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:11.741993 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:11.742310 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:11.742318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:11.742526 (+ 208us) client_negotiation.cc:769] Sending connection context
1210 14:58:11.742691 (+ 165us) client_negotiation.cc:240] Negotiation successful
1210 14:58:11.742929 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20251210 14:58:11.744400 11243 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33513
I20251210 14:58:11.744607 11243 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:11.745054 11243 heartbeater.cc:507] Master 127.6.115.126:33513 requested a full tablet report, sending...
I20251210 14:58:11.745960 10778 ts_manager.cc:194] Registered new tserver with Master: 253a7a34ab9e4df188e60264258cc0cf (127.6.115.67:43901)
I20251210 14:58:11.746449 10778 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:36615
I20251210 14:58:11.748394 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:58:11.771351 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.762915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.6.115.65:35213)
1210 14:58:11.763187 (+ 272us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.763192 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.763213 (+ 21us) server_negotiation.cc:408] Connection header received
1210 14:58:11.763276 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.763280 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.763412 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.763519 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.764259 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.764979 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.765652 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.765905 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.767777 (+ 1872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.767796 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.767805 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.767829 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.769732 (+ 1903us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.770143 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.770151 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.770154 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.770211 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.770442 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.770445 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.770448 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.770634 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.770829 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.771040 (+ 211us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.771177 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:11.780014 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:11.773690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.6.115.126:33513)
1210 14:58:11.773812 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:11.773815 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:11.773830 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:11.773916 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:11.773918 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:11.773952 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:11.774021 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:11.774542 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.774973 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:11.775450 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:11.775552 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:11.777454 (+ 1902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:11.777472 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:11.777474 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:11.777496 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:11.778793 (+ 1297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.779109 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.779111 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.779112 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.779145 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:11.779359 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:11.779361 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:11.779362 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:11.779496 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:11.779588 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:11.779730 (+ 142us) server_negotiation.cc:300] Negotiation successful
1210 14:58:11.779812 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251210 14:58:11.780362 10778 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186
I20251210 14:58:11.780895 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10842
I20251210 14:58:11.787004 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10978
I20251210 14:58:11.792304 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11114
I20251210 14:58:11.797485 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 10746
2025-12-10T14:58:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3041 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[11255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[11255](info): set up 2 sockets
Dec 10 14:58:11 dist-test-slave-nb51 krb5kdc[11255](info): commencing operation
krb5kdc: starting...
W20251210 14:58:13.761765 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:13 dist-test-slave-nb51 krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:13.886477 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:34493
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:13.985105 11271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:13.985284 11271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:13.985384 11271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:13.988552 11271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:13.988610 11271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:13.988632 11271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:13.988651 11271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:13.988669 11271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:13.992708 11271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:13.993682 11271 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:13.994457 11271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:13.999305 11277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:13.999475 11271 server_base.cc:1047] running on GCE node
W20251210 14:58:13.999344 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:13.999577 11276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.000025 11271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:14.000856 11271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:14.002020 11271 hybrid_clock.cc:648] HybridClock initialized: now 1765378694002013 us; error 31 us; skew 500 ppm
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:14.004345 11271 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:14.004926 11271 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:14.004974 11271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:14.005108 11271 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:14.006692 11271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "4c5a67ae61b44e2eb3a638583a4a044f"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "426d46c10da5df3a2c0ce619d2ba0ac7"
server_key_iv: "f8300f166c87232cfa1ae7484e91dc82"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.007115 11271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "4c5a67ae61b44e2eb3a638583a4a044f"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "426d46c10da5df3a2c0ce619d2ba0ac7"
server_key_iv: "f8300f166c87232cfa1ae7484e91dc82"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.010196 11271 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:14.012269 11285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.013216 11271 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:14.013309 11271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "4c5a67ae61b44e2eb3a638583a4a044f"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "426d46c10da5df3a2c0ce619d2ba0ac7"
server_key_iv: "f8300f166c87232cfa1ae7484e91dc82"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.013404 11271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:14.029038 11271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:14.031713 11271 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:14.031857 11271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:14.038313 11337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:34493 every 8 connection(s)
I20251210 14:58:14.038326 11271 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:34493
I20251210 14:58:14.038906 11271 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:14.040759 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11271
I20251210 14:58:14.040877 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:14.041137 6605 external_mini_cluster.cc:1468] Setting key 68476ceb278ff5100626cc33f89020ed
I20251210 14:58:14.041648 11338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:14.046603 11338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: Bootstrap starting.
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:14.048897 11338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:14.049707 11338 log.cc:826] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:14.051395 11338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: No bootstrap required, opened a new log
I20251210 14:58:14.052801 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.042270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.6.115.126:34493)
1210 14:58:14.042611 (+ 341us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.042619 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.042647 (+ 28us) server_negotiation.cc:408] Connection header received
1210 14:58:14.043204 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.043218 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.043464 (+ 246us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.043744 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:14.044612 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.045388 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.045998 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.046270 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.048432 (+ 2162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:14.048449 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:14.048458 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:14.048487 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:14.050699 (+ 2212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.051061 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.051065 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.051069 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.051149 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.051367 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.051370 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.051371 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.051686 (+ 315us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:14.051861 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.052113 (+ 252us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.052325 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":113,"threads_started":1}
I20251210 14:58:14.053761 11338 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } }
I20251210 14:58:14.054000 11338 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:14.054090 11338 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5a67ae61b44e2eb3a638583a4a044f, State: Initialized, Role: FOLLOWER
I20251210 14:58:14.054507 11338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } }
I20251210 14:58:14.054646 11338 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:14.054715 11338 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:14.054805 11338 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:14.055742 11338 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } }
I20251210 14:58:14.056046 11338 leader_election.cc:304] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5a67ae61b44e2eb3a638583a4a044f; no voters:
I20251210 14:58:14.056281 11338 leader_election.cc:290] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:14.056380 11343 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:14.056597 11343 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [term 1 LEADER]: Becoming Leader. State: Replica: 4c5a67ae61b44e2eb3a638583a4a044f, State: Running, Role: LEADER
I20251210 14:58:14.056875 11343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } }
I20251210 14:58:14.057202 11338 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:14.058246 11344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } } }
I20251210 14:58:14.058351 11344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:14.058645 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5a67ae61b44e2eb3a638583a4a044f. Latest consensus state: current_term: 1 leader_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5a67ae61b44e2eb3a638583a4a044f" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34493 } } }
I20251210 14:58:14.058754 11352 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:14.058779 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:14.061293 11352 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:14.065919 11352 catalog_manager.cc:1357] Generated new cluster ID: 226e6853aad240018597a39c883d5413
I20251210 14:58:14.065979 11352 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:14.078027 11352 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:14.079073 11352 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:14.094452 11352 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: Generated new TSK 0
I20251210 14:58:14.095185 11352 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:14.146087 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:34493
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:14.245589 11366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:14.245767 11366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:14.245844 11366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:14.248979 11366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:14.249037 11366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:14.249152 11366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:14.253050 11366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:14.253952 11366 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:14.254678 11366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:14.260317 11374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:14.260380 11372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.260466 11366 server_base.cc:1047] running on GCE node
W20251210 14:58:14.260330 11371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.260898 11366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:14.261399 11366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:14.262605 11366 hybrid_clock.cc:648] HybridClock initialized: now 1765378694262579 us; error 40 us; skew 500 ppm
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:14.265115 11366 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:14.265820 11366 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:14.265884 11366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:14.266047 11366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:14.267613 11366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "5ab5913089f6451c94c3a9d5cf2b1f63"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "5dff1f504549ad7bc95ae7b41956828c"
server_key_iv: "9a3fc202ea953d0db0e4da2cf85c01cc"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.268065 11366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "5ab5913089f6451c94c3a9d5cf2b1f63"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "5dff1f504549ad7bc95ae7b41956828c"
server_key_iv: "9a3fc202ea953d0db0e4da2cf85c01cc"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.271239 11366 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:14.273157 11380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.274088 11366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:14.274209 11366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "5ab5913089f6451c94c3a9d5cf2b1f63"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "5dff1f504549ad7bc95ae7b41956828c"
server_key_iv: "9a3fc202ea953d0db0e4da2cf85c01cc"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.274300 11366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:14.291117 11366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:14.293553 11366 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:14.293715 11366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:14.294188 11366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:14.294945 11366 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:14.295006 11366 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.295066 11366 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:14.295094 11366 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.303331 11366 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:39139
I20251210 14:58:14.303364 11493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:39139 every 8 connection(s)
I20251210 14:58:14.303848 11366 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:14.311192 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11366
I20251210 14:58:14.311308 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:14.311551 6605 external_mini_cluster.cc:1468] Setting key 77d5357a6f638751e370cd9e337ca8a6
I20251210 14:58:14.315316 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.305287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:55311 (local address 127.6.115.126:34493)
1210 14:58:14.305451 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.305455 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.306022 (+ 567us) server_negotiation.cc:408] Connection header received
1210 14:58:14.306843 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.306847 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.306906 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.306977 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:14.308431 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.308966 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.309637 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.309831 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.311839 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:14.311854 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:14.311856 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:14.311878 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:14.313268 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.313851 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.313854 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.313855 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.313891 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.314188 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.314190 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.314191 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.314308 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:14.314375 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.314972 (+ 597us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.315114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251210 14:58:14.315755 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.305522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:34493 (local address 127.6.115.65:55311)
1210 14:58:14.305897 (+ 375us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.305929 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.306653 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.307135 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.307142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.307454 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:14.308288 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.308299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.309087 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.309090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.309522 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.309530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.309708 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.310258 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:14.310292 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:14.311684 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:14.313400 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.313407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.313417 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.313729 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.313977 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.313983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.313987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.314110 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.314396 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:14.314406 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:14.314660 (+ 254us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.314893 (+ 233us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.315092 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20251210 14:58:14.316818 11494 heartbeater.cc:344] Connected to a master server at 127.6.115.126:34493
I20251210 14:58:14.317055 11494 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:14.317548 11494 heartbeater.cc:507] Master 127.6.115.126:34493 requested a full tablet report, sending...
I20251210 14:58:14.318879 11299 ts_manager.cc:194] Registered new tserver with Master: 5ab5913089f6451c94c3a9d5cf2b1f63 (127.6.115.65:39139)
I20251210 14:58:14.320079 11299 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:55311
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:14.356041 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:34493
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:14.453276 11501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:14.453500 11501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:14.453558 11501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:14.456458 11501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:14.456502 11501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:14.456568 11501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:14.460619 11501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:14.461459 11501 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:14.462208 11501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:14.467607 11507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:14.467686 11506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:14.467720 11509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.468046 11501 server_base.cc:1047] running on GCE node
I20251210 14:58:14.468365 11501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:14.468865 11501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:14.470026 11501 hybrid_clock.cc:648] HybridClock initialized: now 1765378694470015 us; error 29 us; skew 500 ppm
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:14.472472 11501 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:14.473129 11501 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:14.473189 11501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:14.473347 11501 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:14.474843 11501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "83e7cf37ac81640b7729e9bcd467a9c3"
server_key_iv: "e940c2c0bc8436439e1bb91735ac3fbf"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.475284 11501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "83e7cf37ac81640b7729e9bcd467a9c3"
server_key_iv: "e940c2c0bc8436439e1bb91735ac3fbf"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.478349 11501 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251210 14:58:14.480479 11515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.481422 11501 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:14.481520 11501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "83e7cf37ac81640b7729e9bcd467a9c3"
server_key_iv: "e940c2c0bc8436439e1bb91735ac3fbf"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.481599 11501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:14.494777 11501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:14.497438 11501 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:14.497613 11501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:14.498126 11501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:14.498860 11501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:14.498903 11501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.498942 11501 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:14.498975 11501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.507225 11501 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:38875
I20251210 14:58:14.507253 11628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:38875 every 8 connection(s)
I20251210 14:58:14.507687 11501 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:14.510283 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11501
I20251210 14:58:14.510396 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:14.510612 6605 external_mini_cluster.cc:1468] Setting key a9cde51d86ab4e215d03c396fe4d83e9
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:58:14.520252 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.509221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:35231 (local address 127.6.115.126:34493)
1210 14:58:14.509413 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.509418 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.509932 (+ 514us) server_negotiation.cc:408] Connection header received
1210 14:58:14.510825 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.510829 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.510887 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.510992 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:14.512790 (+ 1798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.513581 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.514533 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.514660 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.516953 (+ 2293us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:14.516972 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:14.516974 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:14.517002 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:14.518558 (+ 1556us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.519023 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.519027 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.519029 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.519068 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.519332 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.519335 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.519337 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.519467 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:14.519553 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.519991 (+ 438us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.520100 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251210 14:58:14.520897 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.509396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:34493 (local address 127.6.115.66:35231)
1210 14:58:14.509797 (+ 401us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.509828 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.510571 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.511155 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.511167 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.511607 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:14.512635 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.512652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.513725 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.513731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.514401 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.514413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.514598 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.515169 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:14.515220 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:14.516804 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:14.518665 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.518671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.518682 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.518921 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.519167 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.519172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.519174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.519248 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.519563 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:14.519571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:14.519771 (+ 200us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.519940 (+ 169us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.520177 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20251210 14:58:14.522118 11629 heartbeater.cc:344] Connected to a master server at 127.6.115.126:34493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
I20251210 14:58:14.522336 11629 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:14.522753 11629 heartbeater.cc:507] Master 127.6.115.126:34493 requested a full tablet report, sending...
I20251210 14:58:14.523767 11299 ts_manager.cc:194] Registered new tserver with Master: b952e8b02bbb485fb7999ff6ca3ef7c1 (127.6.115.66:38875)
I20251210 14:58:14.524506 11299 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:35231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:14.556079 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:34493
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:14.655913 11636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:14.656087 11636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:14.656140 11636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:14.659128 11636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:14.659183 11636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:14.659262 11636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:14.663112 11636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:14.663990 11636 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:14.664722 11636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:14.670616 11641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.670706 11636 server_base.cc:1047] running on GCE node
W20251210 14:58:14.670631 11642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:14.670627 11644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:14.671183 11636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:14.671694 11636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:14.672869 11636 hybrid_clock.cc:648] HybridClock initialized: now 1765378694672841 us; error 47 us; skew 500 ppm
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:14.675415 11636 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:14.676059 11636 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:14.676119 11636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:14.676280 11636 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:14.677881 11636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "f7cf27ddbcbcb90ea9b2b6b490303c29"
server_key_iv: "b7eaaee437e1d34e696bebc3e5703fb0"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.678341 11636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "f7cf27ddbcbcb90ea9b2b6b490303c29"
server_key_iv: "b7eaaee437e1d34e696bebc3e5703fb0"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.681499 11636 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:14.683346 11650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.684290 11636 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:14.684386 11636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51"
format_stamp: "Formatted at 2025-12-10 14:58:14 on dist-test-slave-nb51"
server_key: "f7cf27ddbcbcb90ea9b2b6b490303c29"
server_key_iv: "b7eaaee437e1d34e696bebc3e5703fb0"
server_key_version: "encryptionkey@0"
I20251210 14:58:14.684463 11636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:14.699592 11636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:14.702497 11636 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:14.702685 11636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:14.703203 11636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:14.704013 11636 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:14.704061 11636 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.704113 11636 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:14.704145 11636 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:14.712787 11636 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:42353
I20251210 14:58:14.712810 11763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:42353 every 8 connection(s)
I20251210 14:58:14.713292 11636 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
Dec 10 14:58:14 dist-test-slave-nb51 krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:14.720883 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11636
I20251210 14:58:14.721000 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:14.721263 6605 external_mini_cluster.cc:1468] Setting key dde50df79696932483989c9eba1a1603
I20251210 14:58:14.725198 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.714811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:41409 (local address 127.6.115.126:34493)
1210 14:58:14.714934 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.714937 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.715542 (+ 605us) server_negotiation.cc:408] Connection header received
1210 14:58:14.716313 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.716317 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.716374 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.716472 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:14.718109 (+ 1637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.718596 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.719224 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.719401 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.721602 (+ 2201us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:14.721618 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:14.721620 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:14.721643 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:14.723567 (+ 1924us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.723994 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.723998 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.723999 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.724040 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.724287 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.724290 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.724292 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.724425 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:14.724511 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.724941 (+ 430us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.725055 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:58:14.725786 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.715041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:34493 (local address 127.6.115.67:41409)
1210 14:58:14.715422 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.715454 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.716137 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.716626 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.716633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.716979 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:14.717975 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.717988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.718709 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.718712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.719114 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.719121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.719301 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.719897 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:14.719918 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:14.721459 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:14.723688 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.723693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.723701 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.723901 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.724116 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:14.724121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:14.724124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:14.724204 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:14.724519 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:14.724527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:14.724723 (+ 196us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.724888 (+ 165us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.725114 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20251210 14:58:14.726649 11764 heartbeater.cc:344] Connected to a master server at 127.6.115.126:34493
I20251210 14:58:14.726873 11764 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:14.727283 11764 heartbeater.cc:507] Master 127.6.115.126:34493 requested a full tablet report, sending...
I20251210 14:58:14.728137 11299 ts_manager.cc:194] Registered new tserver with Master: bbc8d0b1a3204f3ea9e6136bd88cfc51 (127.6.115.67:42353)
I20251210 14:58:14.728715 11299 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:41409
I20251210 14:58:14.734534 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:58:14.742100 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.736165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.6.115.126:34493)
1210 14:58:14.736296 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.736300 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.736406 (+ 106us) server_negotiation.cc:408] Connection header received
1210 14:58:14.736509 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.736512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.736546 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.736608 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:14.737339 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.737832 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.738354 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.738463 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.739414 (+ 951us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:14.739426 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:14.739428 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:14.739449 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:14.740755 (+ 1306us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.741100 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.741103 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.741104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.741143 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:14.741398 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:14.741404 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:14.741408 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:14.741547 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:14.741672 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.741877 (+ 205us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.741958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:14.744688 11299 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:14.747093 11299 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:14.759073 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.755075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:39139 (local address 127.0.0.1:35018)
1210 14:58:14.755906 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.755919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.756003 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.756639 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.756641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.756658 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:14.756878 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.756894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.757789 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.757793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.758594 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.758605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.758712 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.758759 (+ 47us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.758858 (+ 99us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.758947 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":107,"threads_started":1}
I20251210 14:58:14.760003 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.755366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.6.115.67:42353)
1210 14:58:14.756117 (+ 751us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.756125 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.756145 (+ 20us) server_negotiation.cc:408] Connection header received
1210 14:58:14.756234 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.756241 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.756369 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.756495 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:14.756921 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.757993 (+ 1072us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.759149 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.759672 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.759767 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.759826 (+ 59us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.759904 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":135,"threads_started":1}
I20251210 14:58:14.760169 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.755500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.6.115.65:39139)
1210 14:58:14.755924 (+ 424us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.755929 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.756117 (+ 188us) server_negotiation.cc:408] Connection header received
1210 14:58:14.756214 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.756220 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.756355 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.756494 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:14.756995 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.757691 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.759263 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.759785 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.759882 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.759954 (+ 72us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.760037 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":90,"threads_started":1}
I20251210 14:58:14.760280 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.754597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:38875 (local address 127.0.0.1:60670)
1210 14:58:14.754935 (+ 338us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.754976 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.755103 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.756342 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.756348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.756370 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:14.756645 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.756652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.758039 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.758042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.759168 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.759181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.759287 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.759302 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.759350 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.759404 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20251210 14:58:14.759596 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.754763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:42353 (local address 127.0.0.1:57010)
1210 14:58:14.755557 (+ 794us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.755572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.755701 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.756493 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.756495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.756512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:14.756797 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.756804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.758095 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.758099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.759007 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.759016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.759304 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.759328 (+ 24us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.759398 (+ 70us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.759458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":6,"thread_start_us":96,"threads_started":1}
I20251210 14:58:14.760824 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.755252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.6.115.66:38875)
1210 14:58:14.755687 (+ 435us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.755694 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.755720 (+ 26us) server_negotiation.cc:408] Connection header received
1210 14:58:14.755820 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.755827 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.756015 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.756171 (+ 156us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:14.756804 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.757928 (+ 1124us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.759500 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.760015 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.760590 (+ 575us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.760660 (+ 70us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.760727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":141,"threads_started":1}
I20251210 14:58:14.763244 11698 tablet_service.cc:1505] Processing CreateTablet for tablet ea99da81a88140da84bc0ce904b1e02d (DEFAULT_TABLE table=test-table [id=8aaf7daac4d5406fbb1fb26a014363e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:14.763221 11563 tablet_service.cc:1505] Processing CreateTablet for tablet ea99da81a88140da84bc0ce904b1e02d (DEFAULT_TABLE table=test-table [id=8aaf7daac4d5406fbb1fb26a014363e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:14.763219 11428 tablet_service.cc:1505] Processing CreateTablet for tablet ea99da81a88140da84bc0ce904b1e02d (DEFAULT_TABLE table=test-table [id=8aaf7daac4d5406fbb1fb26a014363e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:14.764117 11698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea99da81a88140da84bc0ce904b1e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:14.764133 11428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea99da81a88140da84bc0ce904b1e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:14.764133 11563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea99da81a88140da84bc0ce904b1e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:14.769186 11781 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Bootstrap starting.
I20251210 14:58:14.770422 11782 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Bootstrap starting.
I20251210 14:58:14.770506 11783 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Bootstrap starting.
I20251210 14:58:14.771948 11781 tablet_bootstrap.cc:654] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:14.771991 11782 tablet_bootstrap.cc:654] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:14.772462 11783 tablet_bootstrap.cc:654] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:14.772650 11782 log.cc:826] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:14.772733 11781 log.cc:826] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:14.773329 11783 log.cc:826] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:14.774417 11781 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: No bootstrap required, opened a new log
I20251210 14:58:14.774441 11782 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: No bootstrap required, opened a new log
I20251210 14:58:14.774590 11781 ts_tablet_manager.cc:1403] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251210 14:58:14.774616 11782 ts_tablet_manager.cc:1403] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251210 14:58:14.775122 11783 tablet_bootstrap.cc:492] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: No bootstrap required, opened a new log
I20251210 14:58:14.775286 11783 ts_tablet_manager.cc:1403] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251210 14:58:14.777472 11781 raft_consensus.cc:359] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.777633 11781 raft_consensus.cc:385] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:14.777704 11781 raft_consensus.cc:740] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc8d0b1a3204f3ea9e6136bd88cfc51, State: Initialized, Role: FOLLOWER
I20251210 14:58:14.778129 11781 consensus_queue.cc:260] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.778252 11782 raft_consensus.cc:359] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.778465 11782 raft_consensus.cc:385] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:14.778524 11782 raft_consensus.cc:740] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab5913089f6451c94c3a9d5cf2b1f63, State: Initialized, Role: FOLLOWER
I20251210 14:58:14.778973 11781 ts_tablet_manager.cc:1434] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251210 14:58:14.778908 11783 raft_consensus.cc:359] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.778996 11782 consensus_queue.cc:260] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.779167 11783 raft_consensus.cc:385] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:14.779279 11783 raft_consensus.cc:740] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b952e8b02bbb485fb7999ff6ca3ef7c1, State: Initialized, Role: FOLLOWER
I20251210 14:58:14.779335 11764 heartbeater.cc:499] Master 127.6.115.126:34493 was elected leader, sending a full tablet report...
I20251210 14:58:14.779794 11494 heartbeater.cc:499] Master 127.6.115.126:34493 was elected leader, sending a full tablet report...
I20251210 14:58:14.779750 11783 consensus_queue.cc:260] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.779829 11782 ts_tablet_manager.cc:1434] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251210 14:58:14.780366 11629 heartbeater.cc:499] Master 127.6.115.126:34493 was elected leader, sending a full tablet report...
I20251210 14:58:14.780509 11783 ts_tablet_manager.cc:1434] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20251210 14:58:14.804872 11495 tablet.cc:2378] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:14.958134 11789 raft_consensus.cc:493] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:14.958305 11789 raft_consensus.cc:515] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.959517 11789 leader_election.cc:290] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc8d0b1a3204f3ea9e6136bd88cfc51 (127.6.115.67:42353), 5ab5913089f6451c94c3a9d5cf2b1f63 (127.6.115.65:39139)
I20251210 14:58:14.962218 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.959714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:42353 (local address 127.6.115.66:55373)
1210 14:58:14.959858 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.959873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.959966 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.960266 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.960269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.960288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:14.960506 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.960512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.961223 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.961226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.961831 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.961839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.961948 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.961970 (+ 22us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.962021 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.962079 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251210 14:58:14.962464 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.959793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:39139 (local address 127.6.115.66:42001)
1210 14:58:14.960140 (+ 347us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:14.960150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:14.960250 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:14.960504 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:14.960507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:14.960537 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:14.960796 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.960804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.961496 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.961499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:14.962064 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:14.962071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.962210 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.962227 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:58:14.962278 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:58:14.962329 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20251210 14:58:14.962770 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.959913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:55373 (local address 127.6.115.67:42353)
1210 14:58:14.960044 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.960049 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.960064 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:14.960113 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.960116 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.960169 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.960234 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:14.960638 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.961112 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.961965 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.962484 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.962531 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.962600 (+ 69us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.962663 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251210 14:58:14.962982 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.959888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:42001 (local address 127.6.115.65:39139)
1210 14:58:14.960016 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.960021 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.960226 (+ 205us) server_negotiation.cc:408] Connection header received
1210 14:58:14.960339 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.960344 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.960394 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.960481 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:14.960919 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.961393 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.962201 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.962693 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.962735 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.962784 (+ 49us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.962866 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251210 14:58:14.963320 11718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea99da81a88140da84bc0ce904b1e02d" candidate_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" is_pre_election: true
I20251210 14:58:14.963572 11718 raft_consensus.cc:2468] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b952e8b02bbb485fb7999ff6ca3ef7c1 in term 0.
I20251210 14:58:14.963605 11448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea99da81a88140da84bc0ce904b1e02d" candidate_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" is_pre_election: true
I20251210 14:58:14.963860 11448 raft_consensus.cc:2468] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b952e8b02bbb485fb7999ff6ca3ef7c1 in term 0.
W20251210 14:58:14.964046 11765 tablet.cc:2378] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:14.964047 11519 leader_election.cc:304] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b952e8b02bbb485fb7999ff6ca3ef7c1, bbc8d0b1a3204f3ea9e6136bd88cfc51; no voters:
I20251210 14:58:14.964308 11789 raft_consensus.cc:2804] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:14.964380 11789 raft_consensus.cc:493] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:14.964458 11789 raft_consensus.cc:3060] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:14.965397 11789 raft_consensus.cc:515] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.965723 11789 leader_election.cc:290] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [CANDIDATE]: Term 1 election: Requested vote from peers bbc8d0b1a3204f3ea9e6136bd88cfc51 (127.6.115.67:42353), 5ab5913089f6451c94c3a9d5cf2b1f63 (127.6.115.65:39139)
I20251210 14:58:14.966055 11718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea99da81a88140da84bc0ce904b1e02d" candidate_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51"
I20251210 14:58:14.966111 11448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea99da81a88140da84bc0ce904b1e02d" candidate_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63"
I20251210 14:58:14.966180 11718 raft_consensus.cc:3060] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:14.966262 11448 raft_consensus.cc:3060] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:14.967061 11718 raft_consensus.cc:2468] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b952e8b02bbb485fb7999ff6ca3ef7c1 in term 1.
I20251210 14:58:14.967224 11448 raft_consensus.cc:2468] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b952e8b02bbb485fb7999ff6ca3ef7c1 in term 1.
I20251210 14:58:14.967319 11519 leader_election.cc:304] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b952e8b02bbb485fb7999ff6ca3ef7c1, bbc8d0b1a3204f3ea9e6136bd88cfc51; no voters:
I20251210 14:58:14.967486 11789 raft_consensus.cc:2804] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:14.967717 11789 raft_consensus.cc:697] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 1 LEADER]: Becoming Leader. State: Replica: b952e8b02bbb485fb7999ff6ca3ef7c1, State: Running, Role: LEADER
I20251210 14:58:14.968022 11789 consensus_queue.cc:237] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } }
I20251210 14:58:14.970963 11300 catalog_manager.cc:5654] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b952e8b02bbb485fb7999ff6ca3ef7c1 (127.6.115.66). New cstate: current_term: 1 leader_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b952e8b02bbb485fb7999ff6ca3ef7c1" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 38875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:14.991482 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:14.988601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.6.115.66:38875)
1210 14:58:14.988720 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:14.988724 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:14.988737 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:14.988798 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:14.988800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:14.988866 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:14.988953 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:14.989341 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.989876 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:14.990525 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:14.990649 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:14.990725 (+ 76us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:14.991125 (+ 400us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:14.991234 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:14.991349 (+ 115us) server_negotiation.cc:300] Negotiation successful
1210 14:58:14.991391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251210 14:58:14.996359 11718 raft_consensus.cc:1275] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 1 FOLLOWER]: Refusing update from remote peer b952e8b02bbb485fb7999ff6ca3ef7c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:14.996358 11448 raft_consensus.cc:1275] T ea99da81a88140da84bc0ce904b1e02d P 5ab5913089f6451c94c3a9d5cf2b1f63 [term 1 FOLLOWER]: Refusing update from remote peer b952e8b02bbb485fb7999ff6ca3ef7c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:14.996961 11791 consensus_queue.cc:1048] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab5913089f6451c94c3a9d5cf2b1f63" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:14.997089 11789 consensus_queue.cc:1048] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc8d0b1a3204f3ea9e6136bd88cfc51" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:15.005745 11796 mvcc.cc:204] Tried to move back new op lower bound from 7230991134699630592 to 7230991134592258048. Current Snapshot: MvccSnapshot[applied={T|T < 7230991134699630592}]
I20251210 14:58:15.007136 11795 mvcc.cc:204] Tried to move back new op lower bound from 7230991134699630592 to 7230991134592258048. Current Snapshot: MvccSnapshot[applied={T|T < 7230991134699630592 or (T in {7230991134699630592})}]
I20251210 14:58:15.011408 11300 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56510:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:15.011521 11300 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56510:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:15.013614 11300 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4c5a67ae61b44e2eb3a638583a4a044f: Sending DeleteTablet for 3 replicas of tablet ea99da81a88140da84bc0ce904b1e02d
I20251210 14:58:15.014294 11698 tablet_service.cc:1552] Processing DeleteTablet for tablet ea99da81a88140da84bc0ce904b1e02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:57010
I20251210 14:58:15.014292 11563 tablet_service.cc:1552] Processing DeleteTablet for tablet ea99da81a88140da84bc0ce904b1e02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:60670
I20251210 14:58:15.014777 11805 tablet_replica.cc:333] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: stopping tablet replica
I20251210 14:58:15.014726 11428 tablet_service.cc:1552] Processing DeleteTablet for tablet ea99da81a88140da84bc0ce904b1e02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:35018
I20251210 14:58:15.015034 11805 raft_consensus.cc:2243] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:15.015317 11805 raft_consensus.cc:2272] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:15.015590 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11366
I20251210 14:58:15.017202 11805 ts_tablet_manager.cc:1916] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:15.017501 11804 tablet_replica.cc:333] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: stopping tablet replica
I20251210 14:58:15.017768 11804 raft_consensus.cc:2243] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:15.018081 11804 raft_consensus.cc:2272] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:15.019249 11804 ts_tablet_manager.cc:1916] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:15.019762 11805 ts_tablet_manager.cc:1929] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:15.019876 11805 log.cc:1199] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/ea99da81a88140da84bc0ce904b1e02d
I20251210 14:58:15.020195 11805 ts_tablet_manager.cc:1950] T ea99da81a88140da84bc0ce904b1e02d P bbc8d0b1a3204f3ea9e6136bd88cfc51: Deleting consensus metadata
I20251210 14:58:15.021698 11289 catalog_manager.cc:4985] TS bbc8d0b1a3204f3ea9e6136bd88cfc51 (127.6.115.67:42353): tablet ea99da81a88140da84bc0ce904b1e02d (table test-table [id=8aaf7daac4d5406fbb1fb26a014363e5]) successfully deleted
I20251210 14:58:15.021842 11804 ts_tablet_manager.cc:1929] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:15.021912 11804 log.cc:1199] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/ea99da81a88140da84bc0ce904b1e02d
I20251210 14:58:15.022197 11804 ts_tablet_manager.cc:1950] T ea99da81a88140da84bc0ce904b1e02d P b952e8b02bbb485fb7999ff6ca3ef7c1: Deleting consensus metadata
I20251210 14:58:15.023222 11287 catalog_manager.cc:4985] TS b952e8b02bbb485fb7999ff6ca3ef7c1 (127.6.115.66:38875): tablet ea99da81a88140da84bc0ce904b1e02d (table test-table [id=8aaf7daac4d5406fbb1fb26a014363e5]) successfully deleted
W20251210 14:58:15.025430 11286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:39139 (error 108)
I20251210 14:58:15.025674 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11501
I20251210 14:58:15.027207 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.026882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:39139 (local address 127.0.0.1:35032)
1210 14:58:15.027042 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.027110 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251210 14:58:15.027355 11286 catalog_manager.cc:4712] TS 5ab5913089f6451c94c3a9d5cf2b1f63 (127.6.115.65:39139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea99da81a88140da84bc0ce904b1e02d: Network error: Client connection negotiation failed: client connection to 127.6.115.65:39139: connect: Connection refused (error 111)
I20251210 14:58:15.031607 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11636
I20251210 14:58:15.039228 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11271
2025-12-10T14:58:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3241 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-12-10T14:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:15Z Disabled control of system clock
I20251210 14:58:15.067282 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:33735
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251210 14:58:15.167898 11811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:15.168066 11811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:15.168128 11811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:15.171321 11811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:15.171367 11811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:15.171383 11811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:15.171401 11811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:15.171417 11811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:15.175022 11811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:33735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:15.176100 11811 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:15.177124 11811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:15.182122 11819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.182085 11816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.182076 11817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:15.182412 11811 server_base.cc:1047] running on GCE node
I20251210 14:58:15.182884 11811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:15.183701 11811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:15.184846 11811 hybrid_clock.cc:648] HybridClock initialized: now 1765378695184820 us; error 46 us; skew 500 ppm
I20251210 14:58:15.186429 11811 webserver.cc:492] Webserver started at http://127.6.115.126:39691/ using document root <none> and password file <none>
I20251210 14:58:15.186920 11811 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:15.186978 11811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:15.187155 11811 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:15.188675 11811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "d2fc19e94e924191bae8103e4af6639d"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "dca7a21eda3bbc96638f1743d0191e0a"
server_key_iv: "1e5e97d907250ac9d4303a4588ed3c98"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.189107 11811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "d2fc19e94e924191bae8103e4af6639d"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "dca7a21eda3bbc96638f1743d0191e0a"
server_key_iv: "1e5e97d907250ac9d4303a4588ed3c98"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.192052 11811 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251210 14:58:15.194109 11825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.195067 11811 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:15.195163 11811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "d2fc19e94e924191bae8103e4af6639d"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "dca7a21eda3bbc96638f1743d0191e0a"
server_key_iv: "1e5e97d907250ac9d4303a4588ed3c98"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.195242 11811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:15.199555 11811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:15.200016 11811 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:15.200167 11811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:15.206386 11877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:33735 every 8 connection(s)
I20251210 14:58:15.206384 11811 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:33735
I20251210 14:58:15.207267 11811 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:15.209851 11878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:15.211517 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11811
I20251210 14:58:15.211648 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:15.211925 6605 external_mini_cluster.cc:1468] Setting key f68d8834f01196bc49a53d69fa333420
I20251210 14:58:15.215816 11878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: Bootstrap starting.
I20251210 14:58:15.216939 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.213236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.6.115.126:33735)
1210 14:58:15.213837 (+ 601us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.213845 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.213875 (+ 30us) server_negotiation.cc:408] Connection header received
1210 14:58:15.214380 (+ 505us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.214395 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.214663 (+ 268us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.214934 (+ 271us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.215803 (+ 869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.215808 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.215816 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.215826 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.215864 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.215881 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.216049 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.216195 (+ 146us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.216413 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":170,"threads_started":1}
I20251210 14:58:15.217903 11878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:15.218611 11878 log.cc:826] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:15.220237 11878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: No bootstrap required, opened a new log
I20251210 14:58:15.222738 11878 raft_consensus.cc:359] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } }
I20251210 14:58:15.222908 11878 raft_consensus.cc:385] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:15.222954 11878 raft_consensus.cc:740] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fc19e94e924191bae8103e4af6639d, State: Initialized, Role: FOLLOWER
I20251210 14:58:15.223311 11878 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } }
I20251210 14:58:15.223423 11878 raft_consensus.cc:399] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:15.223475 11878 raft_consensus.cc:493] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:15.223541 11878 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:15.224282 11878 raft_consensus.cc:515] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } }
I20251210 14:58:15.224591 11878 leader_election.cc:304] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fc19e94e924191bae8103e4af6639d; no voters:
I20251210 14:58:15.224835 11878 leader_election.cc:290] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:15.224924 11883 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:15.225126 11883 raft_consensus.cc:697] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [term 1 LEADER]: Becoming Leader. State: Replica: d2fc19e94e924191bae8103e4af6639d, State: Running, Role: LEADER
I20251210 14:58:15.225446 11883 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } }
I20251210 14:58:15.225847 11878 sys_catalog.cc:565] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:15.226895 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fc19e94e924191bae8103e4af6639d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } } }
I20251210 14:58:15.227025 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:15.226895 11885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fc19e94e924191bae8103e4af6639d. Latest consensus state: current_term: 1 leader_uuid: "d2fc19e94e924191bae8103e4af6639d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fc19e94e924191bae8103e4af6639d" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 33735 } } }
I20251210 14:58:15.227125 11885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:15.227406 11892 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:15.230461 11892 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:15.235415 11892 catalog_manager.cc:1357] Generated new cluster ID: cfc77d0504f440229b2cb03017fbacd9
I20251210 14:58:15.235486 11892 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:15.263887 11892 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:15.264204 11892 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:15.273384 11892 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: Generated new TSK 0
I20251210 14:58:15.274008 11892 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:58:15.281348 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--builtin_ntp_servers=127.6.115.84:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251210 14:58:15.411598 11902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:15.411782 11902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:15.411834 11902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:15.414995 11902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:15.415046 11902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:15.415115 11902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:15.418745 11902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:15.419695 11902 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:15.420732 11902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:15.426641 11907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.426678 11908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.426690 11910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:15.426836 11902 server_base.cc:1047] running on GCE node
I20251210 14:58:15.427271 11902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:15.427860 11902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:15.429014 11902 hybrid_clock.cc:648] HybridClock initialized: now 1765378695429007 us; error 26 us; skew 500 ppm
I20251210 14:58:15.430652 11902 webserver.cc:492] Webserver started at http://127.6.115.65:45081/ using document root <none> and password file <none>
I20251210 14:58:15.431161 11902 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:15.431216 11902 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:15.431391 11902 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:15.433003 11902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "8b2bcfbac3f043f38fff202fedc4eebb"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "40d2be685807912edaa21629048fbe03"
server_key_iv: "ee3b7d8dc43f5235319c4530f6428cfd"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.433493 11902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "8b2bcfbac3f043f38fff202fedc4eebb"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "40d2be685807912edaa21629048fbe03"
server_key_iv: "ee3b7d8dc43f5235319c4530f6428cfd"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.436584 11902 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251210 14:58:15.438549 11916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.439445 11902 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:15.439529 11902 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "8b2bcfbac3f043f38fff202fedc4eebb"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "40d2be685807912edaa21629048fbe03"
server_key_iv: "ee3b7d8dc43f5235319c4530f6428cfd"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.439617 11902 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:15.443972 11902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:15.444474 11902 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:15.444615 11902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:15.445066 11902 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:15.445950 11902 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:15.445996 11902 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.446030 11902 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:15.446048 11902 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.454720 11902 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:46763
I20251210 14:58:15.454770 12029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:46763 every 8 connection(s)
I20251210 14:58:15.455513 11902 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:15.456874 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 11902
I20251210 14:58:15.456967 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:15.457211 6605 external_mini_cluster.cc:1468] Setting key 6af89442722dbb04f0883c032ea59429
I20251210 14:58:15.461922 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--builtin_ntp_servers=127.6.115.84:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251210 14:58:15.463188 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.459626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:33497 (local address 127.6.115.126:33735)
1210 14:58:15.459760 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.459763 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.459849 (+ 86us) server_negotiation.cc:408] Connection header received
1210 14:58:15.460928 (+ 1079us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.460932 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.460975 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.461065 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.462528 (+ 1463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.462535 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.462541 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.462583 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.462609 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.462621 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.462766 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.462964 (+ 198us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.463016 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:15.464039 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.458966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33735 (local address 127.6.115.65:33497)
1210 14:58:15.459710 (+ 744us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.459756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.460684 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.461226 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.461237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.461340 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.462254 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.462257 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.462298 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.462360 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.462771 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.462781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.462787 (+ 6us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.463028 (+ 241us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.463229 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
I20251210 14:58:15.465179 12030 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33735
I20251210 14:58:15.465412 12030 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:15.465801 12030 heartbeater.cc:507] Master 127.6.115.126:33735 requested a full tablet report, sending...
I20251210 14:58:15.467350 11842 ts_manager.cc:194] Registered new tserver with Master: 8b2bcfbac3f043f38fff202fedc4eebb (127.6.115.65:46763)
W20251210 14:58:15.564175 12033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:15.564339 12033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:15.564392 12033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:15.567461 12033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:15.567507 12033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:15.567574 12033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:15.571184 12033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:15.572023 12033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:15.573004 12033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:15.578810 12038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.578820 12039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.578851 12041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:15.579037 12033 server_base.cc:1047] running on GCE node
I20251210 14:58:15.579392 12033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:15.579892 12033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:15.581060 12033 hybrid_clock.cc:648] HybridClock initialized: now 1765378695581056 us; error 28 us; skew 500 ppm
I20251210 14:58:15.582738 12033 webserver.cc:492] Webserver started at http://127.6.115.66:35343/ using document root <none> and password file <none>
I20251210 14:58:15.583233 12033 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:15.583289 12033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:15.583451 12033 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:15.584991 12033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "283653f7ec534f0bb217dde75dd4699f"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "9395542a7ef7bdec74499381112c773b"
server_key_iv: "d94400aad76c9a47749d26c5d6860c90"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.585469 12033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "283653f7ec534f0bb217dde75dd4699f"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "9395542a7ef7bdec74499381112c773b"
server_key_iv: "d94400aad76c9a47749d26c5d6860c90"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.588558 12033 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251210 14:58:15.590541 12047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.591506 12033 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:15.591605 12033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "283653f7ec534f0bb217dde75dd4699f"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "9395542a7ef7bdec74499381112c773b"
server_key_iv: "d94400aad76c9a47749d26c5d6860c90"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.591706 12033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:15.595791 12033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:15.596259 12033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:15.596421 12033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:15.596866 12033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:15.597762 12033 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:15.597810 12033 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.597864 12033 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:15.597905 12033 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.607890 12033 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:40831
I20251210 14:58:15.607913 12160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:40831 every 8 connection(s)
I20251210 14:58:15.608994 12033 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:15.613727 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.610685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:50823 (local address 127.6.115.126:33735)
1210 14:58:15.610826 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.610830 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.611543 (+ 713us) server_negotiation.cc:408] Connection header received
1210 14:58:15.612257 (+ 714us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.612260 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.612299 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.612370 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.613317 (+ 947us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.613320 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.613322 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.613332 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.613350 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.613381 (+ 31us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.613470 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.613593 (+ 123us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.613627 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251210 14:58:15.614313 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.610890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33735 (local address 127.6.115.66:50823)
1210 14:58:15.611437 (+ 547us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.611468 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.612092 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.612487 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.612494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.612568 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.613123 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.613124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.613150 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.613196 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.613469 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.613474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.613478 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.613636 (+ 158us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.613778 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20251210 14:58:15.615185 12161 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33735
I20251210 14:58:15.615403 12161 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:15.615744 12161 heartbeater.cc:507] Master 127.6.115.126:33735 requested a full tablet report, sending...
I20251210 14:58:15.616647 11842 ts_manager.cc:194] Registered new tserver with Master: 283653f7ec534f0bb217dde75dd4699f (127.6.115.66:40831)
I20251210 14:58:15.617852 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12033
I20251210 14:58:15.617967 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:15.618222 6605 external_mini_cluster.cc:1468] Setting key b9bf7e0054dd97c65e63b9ab3b065d11
I20251210 14:58:15.619917 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--builtin_ntp_servers=127.6.115.84:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251210 14:58:15.721311 12164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:15.721539 12164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:15.721611 12164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:15.724649 12164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:15.724695 12164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:15.724797 12164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:15.728430 12164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:15.729379 12164 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:15.730412 12164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:15.736191 12169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.736200 12170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:15.736208 12172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:15.736577 12164 server_base.cc:1047] running on GCE node
I20251210 14:58:15.736886 12164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:15.737452 12164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:15.738605 12164 hybrid_clock.cc:648] HybridClock initialized: now 1765378695738593 us; error 30 us; skew 500 ppm
I20251210 14:58:15.740221 12164 webserver.cc:492] Webserver started at http://127.6.115.67:43765/ using document root <none> and password file <none>
I20251210 14:58:15.740739 12164 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:15.740788 12164 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:15.740971 12164 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:15.742534 12164 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "cf4612299e1d42ba96e34862974cb5a5"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "6ddcb9d519aff4af39141698c45c7252"
server_key_iv: "bd28d3d1177424efdb54d3f2b65c2536"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.743000 12164 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "cf4612299e1d42ba96e34862974cb5a5"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "6ddcb9d519aff4af39141698c45c7252"
server_key_iv: "bd28d3d1177424efdb54d3f2b65c2536"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.746124 12164 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:15.748119 12178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.749028 12164 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:15.749115 12164 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "cf4612299e1d42ba96e34862974cb5a5"
format_stamp: "Formatted at 2025-12-10 14:58:15 on dist-test-slave-nb51"
server_key: "6ddcb9d519aff4af39141698c45c7252"
server_key_iv: "bd28d3d1177424efdb54d3f2b65c2536"
server_key_version: "encryptionkey@0"
I20251210 14:58:15.749203 12164 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:15.753379 12164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:15.753844 12164 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:15.753981 12164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:15.754406 12164 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:15.755198 12164 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:15.755239 12164 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.755278 12164 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:15.755330 12164 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:15.764178 12164 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:46001
I20251210 14:58:15.764214 12291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:46001 every 8 connection(s)
I20251210 14:58:15.765014 12164 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:15.769641 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.766548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:54293 (local address 127.6.115.126:33735)
1210 14:58:15.766699 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.766704 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.767483 (+ 779us) server_negotiation.cc:408] Connection header received
1210 14:58:15.768120 (+ 637us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.768123 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.768170 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.768263 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.769221 (+ 958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.769224 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.769226 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.769237 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.769255 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.769263 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.769381 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.769505 (+ 124us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.769541 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251210 14:58:15.770260 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.766810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:33735 (local address 127.6.115.67:54293)
1210 14:58:15.767355 (+ 545us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.767385 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.767967 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.768402 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.768409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.768482 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.769012 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.769014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.769042 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.769090 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.769351 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.769355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.769383 (+ 28us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.769548 (+ 165us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.769704 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20251210 14:58:15.771180 12292 heartbeater.cc:344] Connected to a master server at 127.6.115.126:33735
I20251210 14:58:15.771373 12292 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:15.771690 12292 heartbeater.cc:507] Master 127.6.115.126:33735 requested a full tablet report, sending...
I20251210 14:58:15.772486 11842 ts_manager.cc:194] Registered new tserver with Master: cf4612299e1d42ba96e34862974cb5a5 (127.6.115.67:46001)
I20251210 14:58:15.774590 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12164
I20251210 14:58:15.774681 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:15.774863 6605 external_mini_cluster.cc:1468] Setting key 47f693ff3385de85133e3cb2ee765878
I20251210 14:58:15.776532 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:58:15.779680 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.778206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.6.115.126:33735)
1210 14:58:15.778323 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.778330 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.778484 (+ 154us) server_negotiation.cc:408] Connection header received
1210 14:58:15.778605 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.778607 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.778637 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.778692 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.779392 (+ 700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.779395 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.779397 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.779407 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.779425 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.779432 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.779497 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.779567 (+ 70us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.779599 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:58:15.781961 11842 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:15.784071 11842 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:15.794687 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.791386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.6.115.66:40831)
1210 14:58:15.791730 (+ 344us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.791735 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.791756 (+ 21us) server_negotiation.cc:408] Connection header received
1210 14:58:15.791811 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.791815 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.791962 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.792093 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.793881 (+ 1788us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.793889 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.793900 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.793917 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.793989 (+ 72us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.794010 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.794140 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.794414 (+ 274us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.794500 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20251210 14:58:15.794780 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.791414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:46763 (local address 127.0.0.1:50926)
1210 14:58:15.792225 (+ 811us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.792236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.792314 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.792720 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.792723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.792766 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.793416 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.793420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.793469 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.793529 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.794513 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.794516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.794520 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.794584 (+ 64us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.794658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":88,"threads_started":1}
I20251210 14:58:15.795051 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.791224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40831 (local address 127.0.0.1:59280)
1210 14:58:15.791512 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.791544 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.791622 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.792702 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.792709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.792766 (+ 57us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.793644 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.793646 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.793659 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.793687 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.794192 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.794204 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.794212 (+ 8us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.794871 (+ 659us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.794907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20251210 14:58:15.795099 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.792567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.6.115.67:46001)
1210 14:58:15.793035 (+ 468us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.793040 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.793058 (+ 18us) server_negotiation.cc:408] Connection header received
1210 14:58:15.793119 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.793123 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.793245 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.793407 (+ 162us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.794395 (+ 988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.794399 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.794405 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.794412 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.794437 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.794444 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.794555 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.794901 (+ 346us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.794964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":102,"threads_started":1}
I20251210 14:58:15.795222 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.792228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:46001 (local address 127.0.0.1:33150)
1210 14:58:15.792457 (+ 229us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.792469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.792531 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.793419 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.793422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.793437 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.794010 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.794013 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.794026 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.794049 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.794596 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.794600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.794602 (+ 2us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.794677 (+ 75us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.794700 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20251210 14:58:15.795223 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.791861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.6.115.65:46763)
1210 14:58:15.792229 (+ 368us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.792235 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.792313 (+ 78us) server_negotiation.cc:408] Connection header received
1210 14:58:15.792440 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.792447 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.792601 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.792737 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.794313 (+ 1576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.794322 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.794333 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.794347 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.794401 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.794416 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.794548 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.794602 (+ 54us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.794671 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20251210 14:58:15.797897 12226 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d (DEFAULT_TABLE table=test-table [id=5a5bdaa25df1409a867b2998053972cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:15.797917 11964 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d (DEFAULT_TABLE table=test-table [id=5a5bdaa25df1409a867b2998053972cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:15.797917 12095 tablet_service.cc:1505] Processing CreateTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d (DEFAULT_TABLE table=test-table [id=5a5bdaa25df1409a867b2998053972cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:15.798812 11964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1d2853eea54c7c9459bd94b2228b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:15.798812 12095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1d2853eea54c7c9459bd94b2228b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:15.798817 12226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c1d2853eea54c7c9459bd94b2228b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:15.804280 12309 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Bootstrap starting.
I20251210 14:58:15.804850 12311 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Bootstrap starting.
I20251210 14:58:15.805299 12310 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: Bootstrap starting.
I20251210 14:58:15.806792 12310 tablet_bootstrap.cc:654] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:15.806890 12309 tablet_bootstrap.cc:654] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:15.806890 12311 tablet_bootstrap.cc:654] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:15.807579 12310 log.cc:826] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:15.807744 12311 log.cc:826] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:15.807811 12309 log.cc:826] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:15.809325 12310 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: No bootstrap required, opened a new log
I20251210 14:58:15.809468 12311 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: No bootstrap required, opened a new log
I20251210 14:58:15.809527 12309 tablet_bootstrap.cc:492] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: No bootstrap required, opened a new log
I20251210 14:58:15.809544 12310 ts_tablet_manager.cc:1403] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20251210 14:58:15.809653 12309 ts_tablet_manager.cc:1403] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251210 14:58:15.809653 12311 ts_tablet_manager.cc:1403] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:58:15.811743 12310 raft_consensus.cc:359] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.811929 12310 raft_consensus.cc:385] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:15.812019 12310 raft_consensus.cc:740] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2bcfbac3f043f38fff202fedc4eebb, State: Initialized, Role: FOLLOWER
I20251210 14:58:15.812425 12310 consensus_queue.cc:260] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.812474 12309 raft_consensus.cc:359] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.812484 12311 raft_consensus.cc:359] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.812669 12309 raft_consensus.cc:385] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:15.812669 12311 raft_consensus.cc:385] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:15.812733 12311 raft_consensus.cc:740] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283653f7ec534f0bb217dde75dd4699f, State: Initialized, Role: FOLLOWER
I20251210 14:58:15.812772 12309 raft_consensus.cc:740] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4612299e1d42ba96e34862974cb5a5, State: Initialized, Role: FOLLOWER
I20251210 14:58:15.813118 12030 heartbeater.cc:499] Master 127.6.115.126:33735 was elected leader, sending a full tablet report...
I20251210 14:58:15.813200 12311 consensus_queue.cc:260] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.813323 12309 consensus_queue.cc:260] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.813962 12310 ts_tablet_manager.cc:1434] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20251210 14:58:15.814024 12309 ts_tablet_manager.cc:1434] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251210 14:58:15.814024 12311 ts_tablet_manager.cc:1434] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251210 14:58:15.814332 12292 heartbeater.cc:499] Master 127.6.115.126:33735 was elected leader, sending a full tablet report...
I20251210 14:58:15.814347 12161 heartbeater.cc:499] Master 127.6.115.126:33735 was elected leader, sending a full tablet report...
I20251210 14:58:15.838546 12315 raft_consensus.cc:493] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:15.838703 12315 raft_consensus.cc:515] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.839696 12315 leader_election.cc:290] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283653f7ec534f0bb217dde75dd4699f (127.6.115.66:40831), cf4612299e1d42ba96e34862974cb5a5 (127.6.115.67:46001)
I20251210 14:58:15.841472 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.839990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:44009 (local address 127.6.115.66:40831)
1210 14:58:15.840110 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.840116 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.840128 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:15.840272 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.840275 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.840308 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.840399 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.841059 (+ 660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.841066 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.841072 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.841086 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.841110 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.841120 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.841194 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.841309 (+ 115us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.841343 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251210 14:58:15.841483 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.839889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40831 (local address 127.6.115.65:44009)
1210 14:58:15.840067 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.840080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.840150 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.840422 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.840425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.840440 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.840908 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.840910 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.840919 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.840937 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.841199 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.841204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.841209 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.841290 (+ 81us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.841330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20251210 14:58:15.842093 12115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c1d2853eea54c7c9459bd94b2228b2d" candidate_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283653f7ec534f0bb217dde75dd4699f" is_pre_election: true
I20251210 14:58:15.842341 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.839944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:46001 (local address 127.6.115.65:58499)
1210 14:58:15.840298 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.840308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:15.840368 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:15.840581 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:15.840585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:15.840597 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:15.841782 (+ 1185us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:58:15.841783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:15.841791 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:58:15.841821 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:15.842075 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:15.842079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:15.842084 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:58:15.842166 (+ 82us) client_negotiation.cc:240] Negotiation successful
1210 14:58:15.842197 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20251210 14:58:15.842339 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.840057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:58499 (local address 127.6.115.67:46001)
1210 14:58:15.840176 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.840182 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.840374 (+ 192us) server_negotiation.cc:408] Connection header received
1210 14:58:15.840441 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.840445 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.840482 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.840550 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.841955 (+ 1405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.841957 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.841960 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.841970 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.841987 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.841995 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.842066 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.842194 (+ 128us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.842241 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:58:15.842320 12115 raft_consensus.cc:2468] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2bcfbac3f043f38fff202fedc4eebb in term 0.
I20251210 14:58:15.842823 11919 leader_election.cc:304] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283653f7ec534f0bb217dde75dd4699f, 8b2bcfbac3f043f38fff202fedc4eebb; no voters:
I20251210 14:58:15.842841 12246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c1d2853eea54c7c9459bd94b2228b2d" candidate_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4612299e1d42ba96e34862974cb5a5" is_pre_election: true
I20251210 14:58:15.843026 12246 raft_consensus.cc:2468] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2bcfbac3f043f38fff202fedc4eebb in term 0.
I20251210 14:58:15.843050 12315 raft_consensus.cc:2804] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:15.843113 12315 raft_consensus.cc:493] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:15.843169 12315 raft_consensus.cc:3060] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:15.844306 12315 raft_consensus.cc:515] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.844599 12315 leader_election.cc:290] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [CANDIDATE]: Term 1 election: Requested vote from peers 283653f7ec534f0bb217dde75dd4699f (127.6.115.66:40831), cf4612299e1d42ba96e34862974cb5a5 (127.6.115.67:46001)
I20251210 14:58:15.844941 12246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c1d2853eea54c7c9459bd94b2228b2d" candidate_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4612299e1d42ba96e34862974cb5a5"
I20251210 14:58:15.844941 12115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c1d2853eea54c7c9459bd94b2228b2d" candidate_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283653f7ec534f0bb217dde75dd4699f"
I20251210 14:58:15.845054 12115 raft_consensus.cc:3060] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:15.845054 12246 raft_consensus.cc:3060] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:15.846771 12246 raft_consensus.cc:2468] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2bcfbac3f043f38fff202fedc4eebb in term 1.
I20251210 14:58:15.846771 12115 raft_consensus.cc:2468] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2bcfbac3f043f38fff202fedc4eebb in term 1.
I20251210 14:58:15.847040 11919 leader_election.cc:304] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283653f7ec534f0bb217dde75dd4699f, 8b2bcfbac3f043f38fff202fedc4eebb; no voters:
I20251210 14:58:15.847189 12315 raft_consensus.cc:2804] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:15.847376 12315 raft_consensus.cc:697] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 1 LEADER]: Becoming Leader. State: Replica: 8b2bcfbac3f043f38fff202fedc4eebb, State: Running, Role: LEADER
I20251210 14:58:15.847637 12315 consensus_queue.cc:237] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } }
I20251210 14:58:15.850247 11842 catalog_manager.cc:5654] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2bcfbac3f043f38fff202fedc4eebb (127.6.115.65). New cstate: current_term: 1 leader_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2bcfbac3f043f38fff202fedc4eebb" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 46763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 } health_report { overall_health: UNKNOWN } } }
W20251210 14:58:15.859865 12162 tablet.cc:2378] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:15.872344 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.871046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.6.115.65:46763)
1210 14:58:15.871159 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:15.871162 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:15.871174 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:15.871316 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:15.871318 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:15.871363 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:15.871427 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:15.872051 (+ 624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:15.872054 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:15.872056 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:15.872067 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:15.872087 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:15.872094 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:15.872186 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:15.872230 (+ 44us) server_negotiation.cc:300] Negotiation successful
1210 14:58:15.872261 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251210 14:58:15.877301 12115 raft_consensus.cc:1275] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 1 FOLLOWER]: Refusing update from remote peer 8b2bcfbac3f043f38fff202fedc4eebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:15.877317 12246 raft_consensus.cc:1275] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 1 FOLLOWER]: Refusing update from remote peer 8b2bcfbac3f043f38fff202fedc4eebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:15.877964 12319 consensus_queue.cc:1048] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "283653f7ec534f0bb217dde75dd4699f" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:15.878101 12315 consensus_queue.cc:1048] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4612299e1d42ba96e34862974cb5a5" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:15.885658 12322 mvcc.cc:204] Tried to move back new op lower bound from 7230991138308026368 to 7230991138194837504. Current Snapshot: MvccSnapshot[applied={T|T < 7230991138308026368}]
I20251210 14:58:15.885855 12324 mvcc.cc:204] Tried to move back new op lower bound from 7230991138308026368 to 7230991138194837504. Current Snapshot: MvccSnapshot[applied={T|T < 7230991138308026368 or (T in {7230991138308026368})}]
I20251210 14:58:15.886168 12323 mvcc.cc:204] Tried to move back new op lower bound from 7230991138308026368 to 7230991138194837504. Current Snapshot: MvccSnapshot[applied={T|T < 7230991138308026368}]
I20251210 14:58:15.893119 11842 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42690:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:15.893261 11842 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42690:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:15.895392 11842 catalog_manager.cc:5941] T 00000000000000000000000000000000 P d2fc19e94e924191bae8103e4af6639d: Sending DeleteTablet for 3 replicas of tablet 4c1d2853eea54c7c9459bd94b2228b2d
I20251210 14:58:15.896013 12095 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='slave'} at 127.0.0.1:59280
I20251210 14:58:15.896025 11964 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='slave'} at 127.0.0.1:50926
I20251210 14:58:15.896106 12226 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c1d2853eea54c7c9459bd94b2228b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:15 UTC) from {username='slave'} at 127.0.0.1:33150
I20251210 14:58:15.896497 12332 tablet_replica.cc:333] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb: stopping tablet replica
I20251210 14:58:15.896538 12334 tablet_replica.cc:333] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: stopping tablet replica
I20251210 14:58:15.896771 12332 raft_consensus.cc:2243] T 4c1d2853eea54c7c9459bd94b2228b2d P 8b2bcfbac3f043f38fff202fedc4eebb [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:15.896801 12333 tablet_replica.cc:333] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: stopping tablet replica
I20251210 14:58:15.897024 12333 raft_consensus.cc:2243] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:15.897151 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11902
I20251210 14:58:15.897922 12334 raft_consensus.cc:2243] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:15.898275 12334 raft_consensus.cc:2272] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:15.898795 12333 raft_consensus.cc:2272] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:15.899757 12334 ts_tablet_manager.cc:1916] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:15.901710 12334 ts_tablet_manager.cc:1929] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:15.901779 12334 log.cc:1199] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/4c1d2853eea54c7c9459bd94b2228b2d
I20251210 14:58:15.902035 12334 ts_tablet_manager.cc:1950] T 4c1d2853eea54c7c9459bd94b2228b2d P cf4612299e1d42ba96e34862974cb5a5: Deleting consensus metadata
I20251210 14:58:15.902972 11829 catalog_manager.cc:4985] TS cf4612299e1d42ba96e34862974cb5a5 (127.6.115.67:46001): tablet 4c1d2853eea54c7c9459bd94b2228b2d (table test-table [id=5a5bdaa25df1409a867b2998053972cc]) successfully deleted
W20251210 14:58:15.903666 11826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:46763 (error 108)
I20251210 14:58:15.903833 12333 ts_tablet_manager.cc:1916] T 4c1d2853eea54c7c9459bd94b2228b2d P 283653f7ec534f0bb217dde75dd4699f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:15.904325 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12033
I20251210 14:58:15.904541 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.904280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:46763 (local address 127.0.0.1:50956)
1210 14:58:15.904401 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.904461 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251210 14:58:15.905277 11826 catalog_manager.cc:4712] TS 8b2bcfbac3f043f38fff202fedc4eebb (127.6.115.65:46763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c1d2853eea54c7c9459bd94b2228b2d: Network error: Client connection negotiation failed: client connection to 127.6.115.65:46763: connect: Connection refused (error 111)
I20251210 14:58:15.909952 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12164
I20251210 14:58:15.911054 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:15.910065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40831 (local address 127.0.0.1:59292)
1210 14:58:15.910892 (+ 827us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:15.910958 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.66:40831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704}
I20251210 14:58:15.915246 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 11811
2025-12-10T14:58:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (876 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251210 14:58:15.936908 6605 mini_oidc.cc:150] Starting JWKS server
I20251210 14:58:15.938258 6605 webserver.cc:492] Webserver started at https://127.0.0.1:45779/ using document root <none> and password file <none>
I20251210 14:58:15.938381 6605 mini_oidc.cc:186] Starting OIDC Discovery server
I20251210 14:58:15.938611 6605 webserver.cc:492] Webserver started at http://127.0.0.1:44849/ using document root <none> and password file <none>
2025-12-10T14:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:15Z Disabled control of system clock
I20251210 14:58:15.946352 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43371
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:43371
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45779/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/testchainca.cert with env {}
W20251210 14:58:16.045243 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:16.045429 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:16.045475 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:16.048542 12341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:16.048588 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:16.048605 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:16.048624 12341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:16.048645 12341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251210 14:58:16.048669 12341 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251210 14:58:16.048686 12341 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45779/jwks.json
I20251210 14:58:16.052627 12341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:43371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45779/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:16.053547 12341 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:16.054229 12341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:16.059386 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:16.059417 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:16.059386 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:16.059548 12341 server_base.cc:1047] running on GCE node
I20251210 14:58:16.060140 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:16.060998 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:16.062171 12341 hybrid_clock.cc:648] HybridClock initialized: now 1765378696062156 us; error 30 us; skew 500 ppm
I20251210 14:58:16.063760 12341 webserver.cc:492] Webserver started at http://127.6.115.126:42989/ using document root <none> and password file <none>
I20251210 14:58:16.064266 12341 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:16.064327 12341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:16.064494 12341 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:16.066121 12341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "b29ac033a25d48aca544c5353ed2e4cd"
format_stamp: "Formatted at 2025-12-10 14:58:16 on dist-test-slave-nb51"
server_key: "8490b2550b4a7a4472c189c1d6039678"
server_key_iv: "704550adfde479f9af4536054b5e109b"
server_key_version: "encryptionkey@0"
I20251210 14:58:16.066581 12341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "b29ac033a25d48aca544c5353ed2e4cd"
format_stamp: "Formatted at 2025-12-10 14:58:16 on dist-test-slave-nb51"
server_key: "8490b2550b4a7a4472c189c1d6039678"
server_key_iv: "704550adfde479f9af4536054b5e109b"
server_key_version: "encryptionkey@0"
I20251210 14:58:16.069587 12341 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:16.071621 12355 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:16.072659 12341 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:16.072798 12341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "b29ac033a25d48aca544c5353ed2e4cd"
format_stamp: "Formatted at 2025-12-10 14:58:16 on dist-test-slave-nb51"
server_key: "8490b2550b4a7a4472c189c1d6039678"
server_key_iv: "704550adfde479f9af4536054b5e109b"
server_key_version: "encryptionkey@0"
I20251210 14:58:16.072901 12341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:16.105258 12341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:16.105931 12341 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:16.106109 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:16.112835 12341 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:43371
I20251210 14:58:16.112843 12407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:43371 every 8 connection(s)
I20251210 14:58:16.113863 12341 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:16.116494 12408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:16.120689 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12341
I20251210 14:58:16.120818 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:16.121044 6605 external_mini_cluster.cc:1468] Setting key aeba987f2160506e58eba3ebfc29bc52
I20251210 14:58:16.121654 12408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: Bootstrap starting.
I20251210 14:58:16.123713 12408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:16.124421 12408 log.cc:826] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:16.126062 12408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: No bootstrap required, opened a new log
I20251210 14:58:16.128589 12408 raft_consensus.cc:359] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } }
I20251210 14:58:16.128763 12408 raft_consensus.cc:385] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:16.128841 12408 raft_consensus.cc:740] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29ac033a25d48aca544c5353ed2e4cd, State: Initialized, Role: FOLLOWER
I20251210 14:58:16.129245 12408 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } }
I20251210 14:58:16.129426 12408 raft_consensus.cc:399] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:16.129498 12408 raft_consensus.cc:493] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:16.129611 12408 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:16.130398 12408 raft_consensus.cc:515] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } }
I20251210 14:58:16.130733 12408 leader_election.cc:304] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29ac033a25d48aca544c5353ed2e4cd; no voters:
I20251210 14:58:16.130980 12408 leader_election.cc:290] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:16.131058 12413 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:16.131243 12413 raft_consensus.cc:697] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [term 1 LEADER]: Becoming Leader. State: Replica: b29ac033a25d48aca544c5353ed2e4cd, State: Running, Role: LEADER
I20251210 14:58:16.131366 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:16.122083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.6.115.126:43371)
1210 14:58:16.122668 (+ 585us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:16.122676 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:16.122710 (+ 34us) server_negotiation.cc:408] Connection header received
1210 14:58:16.123225 (+ 515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:16.123239 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:16.123575 (+ 336us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:16.123898 (+ 323us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:16.124478 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.125520 (+ 1042us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:16.126128 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.126413 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:16.129590 (+ 3177us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:16.129611 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:16.129623 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:16.130129 (+ 506us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:16.130170 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:16.130202 (+ 32us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:16.130311 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:16.130473 (+ 162us) server_negotiation.cc:300] Negotiation successful
1210 14:58:16.130719 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20251210 14:58:16.131633 12413 consensus_queue.cc:237] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } }
I20251210 14:58:16.132272 12408 sys_catalog.cc:565] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:16.133037 12415 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b29ac033a25d48aca544c5353ed2e4cd. Latest consensus state: current_term: 1 leader_uuid: "b29ac033a25d48aca544c5353ed2e4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } } }
I20251210 14:58:16.133167 12415 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:16.133594 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b29ac033a25d48aca544c5353ed2e4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29ac033a25d48aca544c5353ed2e4cd" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43371 } } }
I20251210 14:58:16.133683 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:16.134238 6605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20251210 14:58:16.136751 12428 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251210 14:58:16.136816 12428 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251210 14:58:16.136895 12422 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:16.137621 12422 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:16.142393 12422 catalog_manager.cc:1357] Generated new cluster ID: 7a84fbba92974855bfebedd307dcb833
I20251210 14:58:16.142455 12422 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:16.172770 12422 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:16.173637 12422 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:16.185504 12422 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b29ac033a25d48aca544c5353ed2e4cd: Generated new TSK 0
I20251210 14:58:16.186185 12422 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:58:16.227021 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:16.205476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.6.115.126:43371)
1210 14:58:16.205611 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:16.205617 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:16.205777 (+ 160us) server_negotiation.cc:408] Connection header received
1210 14:58:16.205975 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:16.205978 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:16.206025 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:16.206101 (+ 76us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:16.206540 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.207037 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:16.207932 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.208084 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:16.208159 (+ 75us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:16.226591 (+ 18432us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1210 14:58:16.226675 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:16.226839 (+ 164us) server_negotiation.cc:300] Negotiation successful
1210 14:58:16.226885 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"thread_start_us":117,"threads_started":1}
I20251210 14:58:16.237896 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:16.234487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.6.115.126:43371)
1210 14:58:16.234633 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:16.234639 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:16.234874 (+ 235us) server_negotiation.cc:408] Connection header received
1210 14:58:16.234983 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:16.234985 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:16.235022 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:16.235080 (+ 58us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:16.235474 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.235903 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:16.236731 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.236875 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:16.236915 (+ 40us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:16.237648 (+ 733us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1210 14:58:16.237812 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52184: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":56}
W20251210 14:58:16.237952 12411 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52184: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251210 14:58:16.238780 12444 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:16.234528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43371 (local address 127.0.0.1:52184)
1210 14:58:16.234787 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:16.234801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:16.234907 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:16.235101 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:16.235104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:16.235112 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1210 14:58:16.235375 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:16.235382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.236013 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:16.236017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:16.236609 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:16.236618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:16.236722 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:16.236742 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1210 14:58:16.237812 (+ 1070us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1210 14:58:16.237954 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.115.126:43371: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20251210 14:58:31.247191 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:31.243880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.6.115.126:43371)
1210 14:58:31.244179 (+ 299us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.244186 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.244204 (+ 18us) server_negotiation.cc:408] Connection header received
1210 14:58:31.244290 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.244293 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.244338 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.244420 (+ 82us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:31.244795 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.245345 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.246007 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.246201 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.246245 (+ 44us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:31.246877 (+ 632us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1210 14:58:31.247046 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55620: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20251210 14:58:31.247253 12449 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:31.243710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43371 (local address 127.0.0.1:55620)
1210 14:58:31.243992 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:31.244009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:31.244145 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:31.244451 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:31.244454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:31.244462 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1210 14:58:31.244675 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.244682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.245481 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.245484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:31.245879 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.245886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.246013 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.246036 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1210 14:58:31.247000 (+ 964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1210 14:58:31.247122 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.115.126:43371: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
W20251210 14:58:31.247283 12450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55620: JWT verification failed: token expired
W20251210 14:58:31.249512 12450 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:31.248823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.6.115.126:43371)
1210 14:58:31.248939 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.248944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.249013 (+ 69us) server_negotiation.cc:408] Connection header received
1210 14:58:31.249121 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.249124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.249166 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.249230 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:31.249441 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55624: BlockingRecv error: recv got EOF from 127.0.0.1:55624 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251210 14:58:31.253126 12460 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.6.115.126:43371
W20251210 14:58:31.253747 12450 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:31.252867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.6.115.126:43371)
1210 14:58:31.252980 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.252984 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.253069 (+ 85us) server_negotiation.cc:408] Connection header received
1210 14:58:31.253336 (+ 267us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.253340 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.253412 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.253485 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:31.253667 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55626: BlockingRecv error: recv got EOF from 127.0.0.1:55626 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:58:31.253933 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12341
2025-12-10T14:58:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251210 14:58:31.293999 6605 mini_oidc.cc:150] Starting JWKS server
I20251210 14:58:31.294906 6605 webserver.cc:492] Webserver started at https://127.0.0.1:40679/ using document root <none> and password file <none>
I20251210 14:58:31.294991 6605 mini_oidc.cc:186] Starting OIDC Discovery server
I20251210 14:58:31.295207 6605 webserver.cc:492] Webserver started at http://127.0.0.1:40163/ using document root <none> and password file <none>
2025-12-10T14:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:31Z Disabled control of system clock
I20251210 14:58:31.303352 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40539
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:40539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/testchainca.cert with env {}
W20251210 14:58:31.407425 12466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:31.407637 12466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:31.407687 12466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:31.410836 12466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:31.410894 12466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:31.410913 12466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:31.410934 12466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:31.410954 12466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251210 14:58:31.410977 12466 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251210 14:58:31.411002 12466 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40679/jwks.json
I20251210 14:58:31.415028 12466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:40539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:31.415905 12466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:31.416680 12466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:31.422425 12472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:31.422376 12471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:31.422484 12466 server_base.cc:1047] running on GCE node
W20251210 14:58:31.422385 12474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:31.423143 12466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:31.424014 12466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:31.425179 12466 hybrid_clock.cc:648] HybridClock initialized: now 1765378711425158 us; error 43 us; skew 500 ppm
I20251210 14:58:31.426946 12466 webserver.cc:492] Webserver started at http://127.6.115.126:45815/ using document root <none> and password file <none>
I20251210 14:58:31.427524 12466 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:31.427597 12466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:31.427779 12466 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:31.429919 12466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "4cbd60b8d2d14b068007ccb606a9a0a5"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "624f70158dd36fe5d860e37c57513e9c"
server_key_iv: "7895d0f2b0b03437f527164af7e78e78"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.430428 12466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "4cbd60b8d2d14b068007ccb606a9a0a5"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "624f70158dd36fe5d860e37c57513e9c"
server_key_iv: "7895d0f2b0b03437f527164af7e78e78"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.433843 12466 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:31.435982 12480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:31.436990 12466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:31.437108 12466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "4cbd60b8d2d14b068007ccb606a9a0a5"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "624f70158dd36fe5d860e37c57513e9c"
server_key_iv: "7895d0f2b0b03437f527164af7e78e78"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.437198 12466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:31.446870 12466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:31.447420 12466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:31.447595 12466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:31.454689 12466 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:40539
I20251210 14:58:31.454711 12532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:40539 every 8 connection(s)
I20251210 14:58:31.455646 12466 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:31.458246 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12466
I20251210 14:58:31.458364 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:31.458624 6605 external_mini_cluster.cc:1468] Setting key 48655a3fa7f945cff24ac9567d7b14b6
I20251210 14:58:31.459326 12533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:31.467097 12533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5: Bootstrap starting.
I20251210 14:58:31.468576 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:31.459796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.6.115.126:40539)
1210 14:58:31.460384 (+ 588us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.460394 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.460432 (+ 38us) server_negotiation.cc:408] Connection header received
1210 14:58:31.461428 (+ 996us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.461454 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.462159 (+ 705us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.462528 (+ 369us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:31.463611 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.464828 (+ 1217us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.465797 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.466092 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.466688 (+ 596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:31.466736 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:31.466750 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:31.467272 (+ 522us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:31.467323 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:31.467343 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:31.467464 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:31.467609 (+ 145us) server_negotiation.cc:300] Negotiation successful
1210 14:58:31.467839 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20251210 14:58:31.469719 12533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:31.470573 12533 log.cc:826] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:31.471307 6605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251210 14:58:31.472445 12533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5: No bootstrap required, opened a new log
I20251210 14:58:31.474973 12533 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } }
I20251210 14:58:31.475169 12533 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:31.475255 12533 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbd60b8d2d14b068007ccb606a9a0a5, State: Initialized, Role: FOLLOWER
I20251210 14:58:31.475664 12533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } }
I20251210 14:58:31.475795 12533 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:31.475865 12533 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:31.475967 12533 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:31.476826 12533 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } }
I20251210 14:58:31.477147 12533 leader_election.cc:304] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbd60b8d2d14b068007ccb606a9a0a5; no voters:
I20251210 14:58:31.477471 12533 leader_election.cc:290] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:31.477584 12538 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:31.477789 12538 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 4cbd60b8d2d14b068007ccb606a9a0a5, State: Running, Role: LEADER
I20251210 14:58:31.478068 12538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } }
I20251210 14:58:31.478690 12533 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:31.479775 12539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } } }
I20251210 14:58:31.479893 12539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:31.479764 12540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbd60b8d2d14b068007ccb606a9a0a5. Latest consensus state: current_term: 1 leader_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbd60b8d2d14b068007ccb606a9a0a5" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40539 } } }
I20251210 14:58:31.480166 12540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:31.480345 12547 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:31.482785 12547 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:31.487723 12547 catalog_manager.cc:1357] Generated new cluster ID: 945667239c5549ed8b19ffd7feef9f07
I20251210 14:58:31.487779 12547 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:31.501401 12547 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:31.502211 12547 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:31.512480 12547 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4cbd60b8d2d14b068007ccb606a9a0a5: Generated new TSK 0
I20251210 14:58:31.513093 12547 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251210 14:58:31.539338 12563 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251210 14:58:31.539414 12563 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251210 14:58:31.540048 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:31.524735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.6.115.126:40539)
1210 14:58:31.524859 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.524863 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.524960 (+ 97us) server_negotiation.cc:408] Connection header received
1210 14:58:31.525098 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.525101 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.525159 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.525267 (+ 108us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:31.525659 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.526476 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.527083 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.527290 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.527401 (+ 111us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:31.539661 (+ 12260us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1210 14:58:31.539924 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20251210 14:58:31.540073 12562 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:31.524634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40539 (local address 127.0.0.1:32778)
1210 14:58:31.524884 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:31.524896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:31.524983 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:31.525266 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:31.525268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:31.525274 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
1210 14:58:31.525540 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.525550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.526583 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.526586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:31.526991 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.526998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.527089 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.527118 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1210 14:58:31.539844 (+ 12726us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1210 14:58:31.539947 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.115.126:40539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20251210 14:58:31.540122 12536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251210 14:58:31.540953 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12466
2025-12-10T14:58:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (286 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251210 14:58:31.580193 6605 mini_oidc.cc:150] Starting JWKS server
I20251210 14:58:31.581085 6605 webserver.cc:492] Webserver started at https://127.0.0.1:37559/ using document root <none> and password file <none>
I20251210 14:58:31.581219 6605 mini_oidc.cc:186] Starting OIDC Discovery server
I20251210 14:58:31.581525 6605 webserver.cc:492] Webserver started at http://127.0.0.1:45101/ using document root <none> and password file <none>
2025-12-10T14:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:31Z Disabled control of system clock
I20251210 14:58:31.588781 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:34063
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:34063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251210 14:58:31.691874 12569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:31.692106 12569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:31.692194 12569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:31.695400 12569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:31.695465 12569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:31.695494 12569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:31.695525 12569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:31.695567 12569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251210 14:58:31.695608 12569 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251210 14:58:31.695640 12569 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37559/jwks.json
I20251210 14:58:31.699513 12569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:34063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:31.700577 12569 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:31.701720 12569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:31.707208 12574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:31.707259 12577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:31.707448 12569 server_base.cc:1047] running on GCE node
W20251210 14:58:31.707226 12575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:31.707983 12569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:31.708853 12569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:31.710035 12569 hybrid_clock.cc:648] HybridClock initialized: now 1765378711710028 us; error 30 us; skew 500 ppm
I20251210 14:58:31.711791 12569 webserver.cc:492] Webserver started at http://127.6.115.126:36661/ using document root <none> and password file <none>
I20251210 14:58:31.712324 12569 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:31.712378 12569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:31.712581 12569 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:31.714221 12569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "3cc05274a82d4bf6b9fed46fbff31c7c"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "e1620d59f4103cbd67f2c8703b13adbd"
server_key_iv: "25a9e54fbeb2fcc156d5a74ca60fcf98"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.714659 12569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "3cc05274a82d4bf6b9fed46fbff31c7c"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "e1620d59f4103cbd67f2c8703b13adbd"
server_key_iv: "25a9e54fbeb2fcc156d5a74ca60fcf98"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.717859 12569 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:31.719942 12583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:31.720881 12569 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:31.720986 12569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "3cc05274a82d4bf6b9fed46fbff31c7c"
format_stamp: "Formatted at 2025-12-10 14:58:31 on dist-test-slave-nb51"
server_key: "e1620d59f4103cbd67f2c8703b13adbd"
server_key_iv: "25a9e54fbeb2fcc156d5a74ca60fcf98"
server_key_version: "encryptionkey@0"
I20251210 14:58:31.721086 12569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:31.738562 12569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:31.739167 12569 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:31.739331 12569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:31.745959 12569 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:34063
I20251210 14:58:31.745963 12635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:34063 every 8 connection(s)
I20251210 14:58:31.746876 12569 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:31.749536 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:31.753422 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12569
I20251210 14:58:31.753531 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:31.753789 6605 external_mini_cluster.cc:1468] Setting key cb482773de3a16974dd8e25a11398797
I20251210 14:58:31.754899 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c: Bootstrap starting.
I20251210 14:58:31.756989 12636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:31.757747 12636 log.cc:826] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:31.759501 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c: No bootstrap required, opened a new log
I20251210 14:58:31.760828 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:31.754963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.6.115.126:34063)
1210 14:58:31.755544 (+ 581us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.755553 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.755586 (+ 33us) server_negotiation.cc:408] Connection header received
1210 14:58:31.756087 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.756102 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.756433 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.756682 (+ 249us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:31.757242 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.757994 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.758656 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.758926 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.759431 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:31.759450 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:31.759459 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:31.759832 (+ 373us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:31.759870 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:31.759883 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:31.759975 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:31.760119 (+ 144us) server_negotiation.cc:300] Negotiation successful
1210 14:58:31.760323 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20251210 14:58:31.761920 12636 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } }
I20251210 14:58:31.762144 12636 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:31.762202 12636 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc05274a82d4bf6b9fed46fbff31c7c, State: Initialized, Role: FOLLOWER
I20251210 14:58:31.762609 12636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } }
I20251210 14:58:31.762754 12636 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:31.762820 12636 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:31.762912 12636 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:31.763463 6605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251210 14:58:31.763871 12636 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } }
I20251210 14:58:31.764257 12636 leader_election.cc:304] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc05274a82d4bf6b9fed46fbff31c7c; no voters:
I20251210 14:58:31.764559 12636 leader_election.cc:290] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:31.764658 12641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:31.764853 12641 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [term 1 LEADER]: Becoming Leader. State: Replica: 3cc05274a82d4bf6b9fed46fbff31c7c, State: Running, Role: LEADER
I20251210 14:58:31.765138 12641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } }
I20251210 14:58:31.765662 12636 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:31.767346 12643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } } }
I20251210 14:58:31.767467 12643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:31.767730 12642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc05274a82d4bf6b9fed46fbff31c7c. Latest consensus state: current_term: 1 leader_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc05274a82d4bf6b9fed46fbff31c7c" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 34063 } } }
I20251210 14:58:31.768551 12642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:31.768486 12655 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:31.772360 12655 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:31.779536 12655 catalog_manager.cc:1357] Generated new cluster ID: a17f7aaf7ac84ee6813c2739c05c38e6
I20251210 14:58:31.779608 12655 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:31.796202 12655 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:31.797330 12655 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:31.806027 12655 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3cc05274a82d4bf6b9fed46fbff31c7c: Generated new TSK 0
I20251210 14:58:31.806787 12655 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251210 14:58:31.841176 12666 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251210 14:58:31.841256 12666 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251210 14:58:31.842368 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:31.819014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.6.115.126:34063)
1210 14:58:31.819126 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:31.819129 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:31.819310 (+ 181us) server_negotiation.cc:408] Connection header received
1210 14:58:31.819460 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:31.819462 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:31.819522 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:31.819606 (+ 84us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:31.820012 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.820501 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.821308 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.821516 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.821596 (+ 80us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:31.842012 (+ 20416us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1210 14:58:31.842270 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20251210 14:58:31.842443 12639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251210 14:58:31.842509 12665 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:31.818955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:34063 (local address 127.0.0.1:55230)
1210 14:58:31.819238 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:31.819251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:31.819348 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:31.819638 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:31.819641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:31.819650 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1210 14:58:31.819916 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.819923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.820604 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:31.820608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:31.821218 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:31.821228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:31.821335 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:31.821355 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1210 14:58:31.842190 (+ 20835us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1210 14:58:31.842341 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.115.126:34063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20251210 14:58:31.843176 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12569
2025-12-10T14:58:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-12-10T14:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:31Z Disabled control of system clock
I20251210 14:58:31.890192 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39219
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:39219
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/testchainca.cert with env {}
W20251210 14:58:31.991677 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:31.991861 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:31.991921 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:31.994998 12670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:31.995052 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:31.995069 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:31.995087 12670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:31.995105 12670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251210 14:58:31.995122 12670 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251210 14:58:31.995138 12670 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251210 14:58:31.999228 12670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:39219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:32.000204 12670 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:32.000998 12670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:32.006232 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:32.006208 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:32.006229 12678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:32.006642 12670 server_base.cc:1047] running on GCE node
I20251210 14:58:32.007087 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:32.007908 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:32.009081 12670 hybrid_clock.cc:648] HybridClock initialized: now 1765378712009062 us; error 42 us; skew 500 ppm
I20251210 14:58:32.010816 12670 webserver.cc:492] Webserver started at http://127.6.115.126:36685/ using document root <none> and password file <none>
I20251210 14:58:32.011363 12670 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:32.011425 12670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:32.011606 12670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:32.013201 12670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "210e661de66546f49422109449ab54f7"
format_stamp: "Formatted at 2025-12-10 14:58:32 on dist-test-slave-nb51"
server_key: "12f509b651271619b3c54a3b66911c3c"
server_key_iv: "9d2f8b6331534c31cc47e0edd58c8a8c"
server_key_version: "encryptionkey@0"
I20251210 14:58:32.013724 12670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "210e661de66546f49422109449ab54f7"
format_stamp: "Formatted at 2025-12-10 14:58:32 on dist-test-slave-nb51"
server_key: "12f509b651271619b3c54a3b66911c3c"
server_key_iv: "9d2f8b6331534c31cc47e0edd58c8a8c"
server_key_version: "encryptionkey@0"
I20251210 14:58:32.016768 12670 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:58:32.018797 12684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:32.019718 12670 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251210 14:58:32.019822 12670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "210e661de66546f49422109449ab54f7"
format_stamp: "Formatted at 2025-12-10 14:58:32 on dist-test-slave-nb51"
server_key: "12f509b651271619b3c54a3b66911c3c"
server_key_iv: "9d2f8b6331534c31cc47e0edd58c8a8c"
server_key_version: "encryptionkey@0"
I20251210 14:58:32.019903 12670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:32.030455 12670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:32.030936 12670 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:32.031081 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:32.037911 12670 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:39219
I20251210 14:58:32.037971 12736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:39219 every 8 connection(s)
I20251210 14:58:32.038856 12670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:32.041462 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:32.044643 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12670
I20251210 14:58:32.044761 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:32.044976 6605 external_mini_cluster.cc:1468] Setting key 38df239c7b0d3c3399ef60114cbb3616
I20251210 14:58:32.046895 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7: Bootstrap starting.
I20251210 14:58:32.049052 12737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:32.049867 12737 log.cc:826] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:32.051596 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7: No bootstrap required, opened a new log
I20251210 14:58:32.052232 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:32.046203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.6.115.126:39219)
1210 14:58:32.046761 (+ 558us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:32.046769 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:32.046802 (+ 33us) server_negotiation.cc:408] Connection header received
1210 14:58:32.047325 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:32.047342 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:32.047685 (+ 343us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:32.047935 (+ 250us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:32.048480 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.049256 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:32.050129 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.050380 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:32.050869 (+ 489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:32.050885 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:32.050894 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:58:32.051237 (+ 343us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:32.051277 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:58:32.051290 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:32.051387 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:32.051507 (+ 120us) server_negotiation.cc:300] Negotiation successful
1210 14:58:32.051710 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
I20251210 14:58:32.054539 12737 raft_consensus.cc:359] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } }
I20251210 14:58:32.054744 12737 raft_consensus.cc:385] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:32.054800 12737 raft_consensus.cc:740] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210e661de66546f49422109449ab54f7, State: Initialized, Role: FOLLOWER
I20251210 14:58:32.055060 6605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251210 14:58:32.055188 12737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } }
I20251210 14:58:32.055311 12737 raft_consensus.cc:399] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:32.055378 12737 raft_consensus.cc:493] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:32.055469 12737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:32.056288 12737 raft_consensus.cc:515] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } }
I20251210 14:58:32.056639 12737 leader_election.cc:304] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210e661de66546f49422109449ab54f7; no voters:
I20251210 14:58:32.056874 12737 leader_election.cc:290] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:32.056967 12742 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:32.057176 12742 raft_consensus.cc:697] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [term 1 LEADER]: Becoming Leader. State: Replica: 210e661de66546f49422109449ab54f7, State: Running, Role: LEADER
I20251210 14:58:32.057475 12742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } }
I20251210 14:58:32.057938 12737 sys_catalog.cc:565] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:32.059082 12744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210e661de66546f49422109449ab54f7. Latest consensus state: current_term: 1 leader_uuid: "210e661de66546f49422109449ab54f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } } }
I20251210 14:58:32.059203 12744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:32.059190 12743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210e661de66546f49422109449ab54f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e661de66546f49422109449ab54f7" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39219 } } }
I20251210 14:58:32.059253 12743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:32.059510 12751 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:32.062286 12751 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:32.067389 12751 catalog_manager.cc:1357] Generated new cluster ID: 185f8155a3344f1da28517edfb94ac6f
I20251210 14:58:32.067456 12751 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:32.080106 12751 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:32.080928 12751 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:32.094900 12751 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 210e661de66546f49422109449ab54f7: Generated new TSK 0
I20251210 14:58:32.095573 12751 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:58:32.125203 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:32.102966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.6.115.126:39219)
1210 14:58:32.103109 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:32.103113 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:32.103263 (+ 150us) server_negotiation.cc:408] Connection header received
1210 14:58:32.103481 (+ 218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:32.103484 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:32.103537 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:32.103631 (+ 94us) server_negotiation.cc:227] Negotiated authn=JWT
1210 14:58:32.104047 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.104566 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:32.105487 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.105653 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:32.105732 (+ 79us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1210 14:58:32.124827 (+ 19095us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1210 14:58:32.125074 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20251210 14:58:32.125272 12740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251210 14:58:32.125694 12766 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:32.102886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39219 (local address 127.0.0.1:58974)
1210 14:58:32.103173 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:32.103189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:32.103294 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:32.103649 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:32.103652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:32.103661 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1210 14:58:32.103931 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:32.103952 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.104685 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:32.104689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:32.105379 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:32.105391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:32.105525 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:32.105547 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1210 14:58:32.125031 (+ 19484us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1210 14:58:32.125204 (+ 173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.115.126:39219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20251210 14:58:32.126530 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12670
2025-12-10T14:58:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (262 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (112 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (109 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:32 dist-test-slave-nb51 krb5kdc[12778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:32 dist-test-slave-nb51 krb5kdc[12778](info): set up 2 sockets
Dec 10 14:58:32 dist-test-slave-nb51 krb5kdc[12778](info): commencing operation
krb5kdc: starting...
W20251210 14:58:34.293778 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.926s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:34 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:34.422132 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45441
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:45441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:34.522308 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:34.522501 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:34.522578 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:34.525709 12794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:34.525760 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:34.525799 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:34.525839 12794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:34.525885 12794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:34.529886 12794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:45441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:34.530838 12794 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:34.531651 12794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:34.536717 12799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:34.536720 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:34.536679 12802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:34.536867 12794 server_base.cc:1047] running on GCE node
I20251210 14:58:34.537484 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:34.538319 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:34.539462 12794 hybrid_clock.cc:648] HybridClock initialized: now 1765378714539467 us; error 48 us; skew 500 ppm
Dec 10 14:58:34 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378714, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:34.541903 12794 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:34.542814 12794 webserver.cc:492] Webserver started at http://127.6.115.126:40347/ using document root <none> and password file <none>
I20251210 14:58:34.543279 12794 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:34.543324 12794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:34.543465 12794 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:34.545023 12794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "b4bb0b79c5364a1a829cd7923448e518"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "350b76f183178908d63ce4862b256c01"
server_key_iv: "db1b4ed00df5a20cef7563bc4033f0bd"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.545511 12794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "b4bb0b79c5364a1a829cd7923448e518"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "350b76f183178908d63ce4862b256c01"
server_key_iv: "db1b4ed00df5a20cef7563bc4033f0bd"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.548483 12794 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251210 14:58:34.550555 12809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:34.551405 12794 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:34.551484 12794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "b4bb0b79c5364a1a829cd7923448e518"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "350b76f183178908d63ce4862b256c01"
server_key_iv: "db1b4ed00df5a20cef7563bc4033f0bd"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.551546 12794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:34.592700 12794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:34.598019 12794 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:34.598162 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:34.605077 12794 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:45441
I20251210 14:58:34.605077 12861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:45441 every 8 connection(s)
I20251210 14:58:34.606089 12794 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:34.607270 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12794
I20251210 14:58:34.607374 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:34.607599 6605 external_mini_cluster.cc:1468] Setting key 1f215cdba93da322fc16ceac010f462b
I20251210 14:58:34.608966 12862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:34.614780 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518: Bootstrap starting.
Dec 10 14:58:34 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:34.616963 12862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:34.617810 12862 log.cc:826] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:34.619522 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518: No bootstrap required, opened a new log
I20251210 14:58:34.620718 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:34.608721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.6.115.126:45441)
1210 14:58:34.609111 (+ 390us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:34.609122 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:34.609161 (+ 39us) server_negotiation.cc:408] Connection header received
1210 14:58:34.610603 (+ 1442us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:34.610627 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:34.610999 (+ 372us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:34.611379 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:34.612120 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.613189 (+ 1069us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:34.613854 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.614121 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:34.616152 (+ 2031us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:34.616195 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:34.616207 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:34.616247 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:34.618275 (+ 2028us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:34.618695 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:34.618699 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:34.618703 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:34.618769 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:34.618997 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:34.619000 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:34.619002 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:34.619291 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:34.619514 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:34.619787 (+ 273us) server_negotiation.cc:300] Negotiation successful
1210 14:58:34.620039 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20251210 14:58:34.622643 12862 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } }
I20251210 14:58:34.622812 12862 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:34.622867 12862 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bb0b79c5364a1a829cd7923448e518, State: Initialized, Role: FOLLOWER
I20251210 14:58:34.623250 12862 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } }
I20251210 14:58:34.623363 12862 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:34.623425 12862 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:34.623528 12862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:34.624672 12862 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } }
I20251210 14:58:34.624985 12862 leader_election.cc:304] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bb0b79c5364a1a829cd7923448e518; no voters:
I20251210 14:58:34.625227 12862 leader_election.cc:290] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:34.625303 12867 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:34.625535 12867 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [term 1 LEADER]: Becoming Leader. State: Replica: b4bb0b79c5364a1a829cd7923448e518, State: Running, Role: LEADER
I20251210 14:58:34.626076 12867 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } }
I20251210 14:58:34.626307 12862 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:34.627665 12869 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bb0b79c5364a1a829cd7923448e518. Latest consensus state: current_term: 1 leader_uuid: "b4bb0b79c5364a1a829cd7923448e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } } }
I20251210 14:58:34.627717 12868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bb0b79c5364a1a829cd7923448e518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bb0b79c5364a1a829cd7923448e518" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45441 } } }
I20251210 14:58:34.627763 12869 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:34.627830 12868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:34.628166 12876 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:34.631045 12876 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:34.636122 12876 catalog_manager.cc:1357] Generated new cluster ID: 5ca6739fd703424fb827a5492db6128f
I20251210 14:58:34.636186 12876 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:34.656772 12876 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:34.657716 12876 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:34.672348 12876 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b4bb0b79c5364a1a829cd7923448e518: Generated new TSK 0
I20251210 14:58:34.673022 12876 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:34.726688 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45441
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:34.850265 12890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:34.850435 12890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:34.850481 12890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:34.853603 12890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:34.853660 12890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:34.853744 12890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:34.857524 12890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:34.858397 12890 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:34.859113 12890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:34.865126 12895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:34.865098 12898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:34.865129 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:34.865382 12890 server_base.cc:1047] running on GCE node
I20251210 14:58:34.865770 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:34.866245 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:34.867384 12890 hybrid_clock.cc:648] HybridClock initialized: now 1765378714867366 us; error 38 us; skew 500 ppm
Dec 10 14:58:34 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378714, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:34.869648 12890 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:34.870702 12890 webserver.cc:492] Webserver started at http://127.6.115.65:38899/ using document root <none> and password file <none>
I20251210 14:58:34.871197 12890 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:34.871250 12890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:34.871413 12890 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:34.872947 12890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "99ce62a4dce94a60b1e503b3d24e7f0b"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "35caa5a2ff729278077beeed8ceda0f9"
server_key_iv: "aff8989a152abf8bcfc7761429ae13e8"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.873406 12890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "99ce62a4dce94a60b1e503b3d24e7f0b"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "35caa5a2ff729278077beeed8ceda0f9"
server_key_iv: "aff8989a152abf8bcfc7761429ae13e8"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.876487 12890 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:58:34.878479 12905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:34.879411 12890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251210 14:58:34.879504 12890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "99ce62a4dce94a60b1e503b3d24e7f0b"
format_stamp: "Formatted at 2025-12-10 14:58:34 on dist-test-slave-nb51"
server_key: "35caa5a2ff729278077beeed8ceda0f9"
server_key_iv: "aff8989a152abf8bcfc7761429ae13e8"
server_key_version: "encryptionkey@0"
I20251210 14:58:34.879607 12890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:34.893298 12890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:34.896267 12890 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:34.896446 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:34.897038 12890 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:34.897900 12890 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:34.897948 12890 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:34.897986 12890 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:34.898021 12890 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:34.906575 12890 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:40651
I20251210 14:58:34.906600 13018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:40651 every 8 connection(s)
I20251210 14:58:34.907473 12890 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:34.912551 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 12890
I20251210 14:58:34.912685 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:34.912926 6605 external_mini_cluster.cc:1468] Setting key 1fe08f88d558b8522d51c4c7a6c78ad3
Dec 10 14:58:34 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378714, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:34.920663 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:34.909243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:53989 (local address 127.6.115.126:45441)
1210 14:58:34.909407 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:34.909413 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:34.910031 (+ 618us) server_negotiation.cc:408] Connection header received
1210 14:58:34.910845 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:34.910848 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:34.910905 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:34.910993 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:34.912522 (+ 1529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.913137 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:34.913851 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.914030 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:34.916823 (+ 2793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:34.916848 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:34.916854 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:34.916886 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:34.918724 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:34.919217 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:34.919223 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:34.919227 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:34.919286 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:34.919547 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:34.919552 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:34.919556 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:34.919782 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:34.919897 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:34.920324 (+ 427us) server_negotiation.cc:300] Negotiation successful
1210 14:58:34.920472 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:34.921182 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:34.909478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45441 (local address 127.6.115.65:53989)
1210 14:58:34.909903 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:34.909935 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:34.910659 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:34.911148 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:34.911156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:34.911500 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:34.912386 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:34.912398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.913249 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:34.913252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:34.913737 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:34.913745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:34.913951 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:34.914539 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:34.914570 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:34.916656 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:34.918848 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:34.918852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:34.918860 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:34.919113 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:34.919389 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:34.919391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:34.919392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:34.919461 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:34.919892 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:34.919897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:34.920094 (+ 197us) client_negotiation.cc:769] Sending connection context
1210 14:58:34.920253 (+ 159us) client_negotiation.cc:240] Negotiation successful
1210 14:58:34.920449 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:34.922281 13019 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45441
I20251210 14:58:34.922537 13019 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:34.923038 13019 heartbeater.cc:507] Master 127.6.115.126:45441 requested a full tablet report, sending...
I20251210 14:58:34.924436 12826 ts_manager.cc:194] Registered new tserver with Master: 99ce62a4dce94a60b1e503b3d24e7f0b (127.6.115.65:40651)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:58:34.925750 12826 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:53989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:34.960173 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45441
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:35.062194 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:35.062395 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:35.062446 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:35.065575 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:35.065644 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:35.065718 13026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:35.069511 13026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:35.070433 13026 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:35.071194 13026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:35.077143 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:35.077148 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:35.077148 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:35.077286 13026 server_base.cc:1047] running on GCE node
I20251210 14:58:35.077726 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:35.078230 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:35.079368 13026 hybrid_clock.cc:648] HybridClock initialized: now 1765378715079354 us; error 27 us; skew 500 ppm
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.081815 13026 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:35.082754 13026 webserver.cc:492] Webserver started at http://127.6.115.66:41889/ using document root <none> and password file <none>
I20251210 14:58:35.083259 13026 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:35.083312 13026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:35.083496 13026 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:35.085172 13026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "c1782c7701004dff9987b923d1f562ae"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "2f9772813016a483a667cc2a61c7fc34"
server_key_iv: "e8f79e4d7e5bfe6cd9fdaa63e431d48b"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.085672 13026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "c1782c7701004dff9987b923d1f562ae"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "2f9772813016a483a667cc2a61c7fc34"
server_key_iv: "e8f79e4d7e5bfe6cd9fdaa63e431d48b"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.088651 13026 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:35.090687 13041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.091557 13026 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:35.091677 13026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "c1782c7701004dff9987b923d1f562ae"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "2f9772813016a483a667cc2a61c7fc34"
server_key_iv: "e8f79e4d7e5bfe6cd9fdaa63e431d48b"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.091755 13026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:35.104285 13026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:35.106935 13026 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:35.107074 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:35.107504 13026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:35.108290 13026 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:35.108335 13026 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.108371 13026 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:35.108390 13026 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.116607 13026 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:40131
I20251210 14:58:35.116667 13154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:40131 every 8 connection(s)
I20251210 14:58:35.117496 13026 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:35.125543 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13026
I20251210 14:58:35.125661 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:35.125885 6605 external_mini_cluster.cc:1468] Setting key 05bd58ab1a3c8ea98c4de6004bedd61e
I20251210 14:58:35.129689 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.119075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40299 (local address 127.6.115.126:45441)
1210 14:58:35.119210 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.119215 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.119871 (+ 656us) server_negotiation.cc:408] Connection header received
1210 14:58:35.120815 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.120818 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.120866 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.120940 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.122486 (+ 1546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.123010 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.123600 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.123724 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.125809 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:35.125827 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:35.125830 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:35.125856 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:35.127611 (+ 1755us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.128241 (+ 630us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.128244 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.128245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.128293 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.128629 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.128632 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.128633 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.128780 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:35.128900 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:35.129357 (+ 457us) server_negotiation.cc:300] Negotiation successful
1210 14:58:35.129504 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:58:35.130304 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.119337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45441 (local address 127.6.115.66:40299)
1210 14:58:35.119744 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:35.119779 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:35.120612 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:35.121070 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:35.121077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:35.121468 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:35.122353 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:35.122366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.123103 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.123105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:35.123516 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:35.123522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.123689 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.124254 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:35.124275 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:35.125652 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:35.127728 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:35.127736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:35.127750 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:35.128140 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:35.128386 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:35.128392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:35.128397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:35.128514 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:35.128914 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:35.128923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:35.129128 (+ 205us) client_negotiation.cc:769] Sending connection context
1210 14:58:35.129306 (+ 178us) client_negotiation.cc:240] Negotiation successful
1210 14:58:35.129598 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20251210 14:58:35.131348 13155 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45441
I20251210 14:58:35.131589 13155 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:35.132054 13155 heartbeater.cc:507] Master 127.6.115.126:45441 requested a full tablet report, sending...
I20251210 14:58:35.133056 12826 ts_manager.cc:194] Registered new tserver with Master: c1782c7701004dff9987b923d1f562ae (127.6.115.66:40131)
I20251210 14:58:35.133900 12826 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:40299
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:35.170485 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45441
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:35.270308 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:35.270493 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:35.270572 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:35.273623 13162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:35.273681 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:35.273777 13162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:35.277719 13162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:35.278702 13162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:35.279431 13162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:35.285534 13168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:35.285584 13170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:35.285684 13162 server_base.cc:1047] running on GCE node
W20251210 14:58:35.285540 13167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:35.286152 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:35.286662 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:35.287796 13162 hybrid_clock.cc:648] HybridClock initialized: now 1765378715287774 us; error 37 us; skew 500 ppm
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.290041 13162 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:35.290971 13162 webserver.cc:492] Webserver started at http://127.6.115.67:46027/ using document root <none> and password file <none>
I20251210 14:58:35.291455 13162 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:35.291505 13162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:35.291679 13162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:35.293179 13162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "085791ac2bc44966b55b5f0e9533b7ce"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "98432e39bd4d5a92b383c593631f10aa"
server_key_iv: "91cfe998ced51b1e6e3f4de9771c59e6"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.293620 13162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "085791ac2bc44966b55b5f0e9533b7ce"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "98432e39bd4d5a92b383c593631f10aa"
server_key_iv: "91cfe998ced51b1e6e3f4de9771c59e6"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.296780 13162 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:35.298775 13177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.299664 13162 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:35.299758 13162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "085791ac2bc44966b55b5f0e9533b7ce"
format_stamp: "Formatted at 2025-12-10 14:58:35 on dist-test-slave-nb51"
server_key: "98432e39bd4d5a92b383c593631f10aa"
server_key_iv: "91cfe998ced51b1e6e3f4de9771c59e6"
server_key_version: "encryptionkey@0"
I20251210 14:58:35.299839 13162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:35.317647 13162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:35.320497 13162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:35.320672 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:35.321189 13162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:35.322003 13162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:35.322048 13162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.322110 13162 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:35.322135 13162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:35.331286 13162 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:33241
I20251210 14:58:35.331311 13290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:33241 every 8 connection(s)
I20251210 14:58:35.332162 13162 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:35.335443 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13162
I20251210 14:58:35.335582 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:35.335897 6605 external_mini_cluster.cc:1468] Setting key b2690413976770b899a9efb949353a80
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:35.344451 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.334001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:36919 (local address 127.6.115.126:45441)
1210 14:58:35.334144 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.334149 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.334848 (+ 699us) server_negotiation.cc:408] Connection header received
1210 14:58:35.335901 (+ 1053us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.335904 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.335956 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.336064 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.338104 (+ 2040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.338588 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.339271 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.339413 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.341441 (+ 2028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:35.341454 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:35.341457 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:35.341480 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:35.342823 (+ 1343us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.343271 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.343273 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.343274 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.343308 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.343567 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.343570 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.343572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.343710 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:35.343814 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:35.344249 (+ 435us) server_negotiation.cc:300] Negotiation successful
1210 14:58:35.344336 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:58:35.345088 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.334242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45441 (local address 127.6.115.67:36919)
1210 14:58:35.334702 (+ 460us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:35.334742 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:35.335682 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:35.336253 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:35.336261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:35.336849 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:35.337973 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:35.337991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.338702 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.338705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:35.339152 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:35.339159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.339331 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.339894 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:35.339917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:35.341281 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:35.342944 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:35.342948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:35.342956 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:35.343191 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:35.343379 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:35.343381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:35.343383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:35.343455 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:35.343828 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:35.343834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:35.344041 (+ 207us) client_negotiation.cc:769] Sending connection context
1210 14:58:35.344214 (+ 173us) client_negotiation.cc:240] Negotiation successful
1210 14:58:35.344409 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
I20251210 14:58:35.346071 13291 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45441
I20251210 14:58:35.346284 13291 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:35.346763 13291 heartbeater.cc:507] Master 127.6.115.126:45441 requested a full tablet report, sending...
I20251210 14:58:35.347623 12826 ts_manager.cc:194] Registered new tserver with Master: 085791ac2bc44966b55b5f0e9533b7ce (127.6.115.67:33241)
I20251210 14:58:35.348142 12826 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:36919
I20251210 14:58:35.349545 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/test-admin.keytab.
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.365096 6605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251210 14:58:35.367897 13298 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:35.366541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.6.115.65:40651)
1210 14:58:35.366857 (+ 316us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.366864 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.366888 (+ 24us) server_negotiation.cc:408] Connection header received
1210 14:58:35.366941 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.366946 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.367106 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.367235 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.367675 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54586: BlockingRecv error: recv got EOF from 127.0.0.1:54586 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
W20251210 14:58:35.367978 6605 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc' of type FILE
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.369225 6605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.370227 6605 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:58:35.378777 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.371144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.6.115.65:40651)
1210 14:58:35.371255 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.371258 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.371272 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:58:35.371359 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.371362 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.371404 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.371477 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.372069 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.372783 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.373352 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.373573 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.375478 (+ 1905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:35.375501 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:35.375516 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:35.375541 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:35.377193 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.377619 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.377624 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.377626 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.377681 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.377869 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.377871 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.377873 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.378048 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:35.378277 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:35.378427 (+ 150us) server_negotiation.cc:300] Negotiation successful
1210 14:58:35.378602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20251210 14:58:35.382104 13298 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:35.381242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.6.115.65:40651)
1210 14:58:35.381400 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.381405 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.381465 (+ 60us) server_negotiation.cc:408] Connection header received
1210 14:58:35.381583 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.381587 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.381632 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.381696 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.382015 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54602: BlockingRecv error: recv got EOF from 127.0.0.1:54602 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20251210 14:58:35.382241 6605 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc' of type FILE
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.383167 6605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.384093 6605 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:58:35.391517 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.384848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.6.115.65:40651)
1210 14:58:35.384955 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.384958 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.384993 (+ 35us) server_negotiation.cc:408] Connection header received
1210 14:58:35.385089 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.385091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.385126 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.385190 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.385756 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.386210 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.386784 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.386935 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.388744 (+ 1809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:35.388759 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:35.388761 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:35.388783 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:35.390228 (+ 1445us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.390561 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.390564 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.390565 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.390609 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.390815 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.390818 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.390820 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.390944 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:35.391066 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:35.391230 (+ 164us) server_negotiation.cc:300] Negotiation successful
1210 14:58:35.391330 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251210 14:58:35.393905 13298 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:35.393107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.6.115.65:40651)
1210 14:58:35.393221 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.393224 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.393285 (+ 61us) server_negotiation.cc:408] Connection header received
1210 14:58:35.393407 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.393410 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.393449 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.393512 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.393825 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54620: BlockingRecv error: recv got EOF from 127.0.0.1:54620 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251210 14:58:35.394050 6605 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765378657750705-6605-0/krb5kdc/krb5cc' of type FILE
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.395001 6605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:35.395933 6605 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[12778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.65@KRBTEST.COM
I20251210 14:58:35.403281 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:35.396711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.6.115.65:40651)
1210 14:58:35.396840 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:35.396843 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:35.396907 (+ 64us) server_negotiation.cc:408] Connection header received
1210 14:58:35.396998 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:35.397000 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:35.397034 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:35.397097 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:35.397702 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.398149 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:35.398708 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:35.398828 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:35.400732 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:35.400748 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:35.400750 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:35.400772 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:35.402195 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.402549 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.402551 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.402553 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.402586 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:35.402736 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:35.402739 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:35.402741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:35.402844 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:35.402931 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:35.403097 (+ 166us) server_negotiation.cc:300] Negotiation successful
1210 14:58:35.403178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251210 14:58:35.404263 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12890
I20251210 14:58:35.410244 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13026
I20251210 14:58:35.415709 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13162
I20251210 14:58:35.420966 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 12794
2025-12-10T14:58:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3075 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[13311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[13311](info): set up 2 sockets
Dec 10 14:58:35 dist-test-slave-nb51 krb5kdc[13311](info): commencing operation
krb5kdc: starting...
W20251210 14:58:37.393982 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.950s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:37 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:37.521270 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45729
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:45729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:37.623561 13327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:37.623769 13327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:37.623821 13327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:37.626947 13327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:37.627015 13327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:37.627038 13327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:37.627058 13327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:37.627074 13327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:37.631107 13327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:45729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:37.632036 13327 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:37.632787 13327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:37.637811 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:37.637820 13335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:37.637820 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:37.638128 13327 server_base.cc:1047] running on GCE node
I20251210 14:58:37.638655 13327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:37.639500 13327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:37.640645 13327 hybrid_clock.cc:648] HybridClock initialized: now 1765378717640622 us; error 35 us; skew 500 ppm
Dec 10 14:58:37 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:37.643183 13327 init.cc:377] Logged in from keytab as oryx/127.6.115.126@KRBTEST.COM (short username oryx)
I20251210 14:58:37.644165 13327 webserver.cc:492] Webserver started at http://127.6.115.126:43623/ using document root <none> and password file <none>
I20251210 14:58:37.644670 13327 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:37.644714 13327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:37.644860 13327 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:37.646566 13327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "15073e88db734900ac12653cfa564e88"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "1d3515bc9fd8c9057aaf96affd274dfa"
server_key_iv: "1ff252ee3f00f8f698d7f2dd2db813d0"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.647010 13327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "15073e88db734900ac12653cfa564e88"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "1d3515bc9fd8c9057aaf96affd274dfa"
server_key_iv: "1ff252ee3f00f8f698d7f2dd2db813d0"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.650141 13327 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:37.652197 13342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:37.653085 13327 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:37.653172 13327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "15073e88db734900ac12653cfa564e88"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "1d3515bc9fd8c9057aaf96affd274dfa"
server_key_iv: "1ff252ee3f00f8f698d7f2dd2db813d0"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.653237 13327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:37.671927 13327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:37.674590 13327 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:37.674736 13327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:37.681720 13327 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:45729
I20251210 14:58:37.681723 13394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:45729 every 8 connection(s)
I20251210 14:58:37.682647 13327 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:37.685473 13395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:37.686805 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13327
I20251210 14:58:37.686944 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:37.687217 6605 external_mini_cluster.cc:1468] Setting key 371f3f96b5f2e32f5085bc85d70d67d0
I20251210 14:58:37.691622 13395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Bootstrap starting.
I20251210 14:58:37.693996 13395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:37.694803 13395 log.cc:826] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Log is configured to *not* fsync() on all Append() calls
Dec 10 14:58:37 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:37.696746 13395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: No bootstrap required, opened a new log
I20251210 14:58:37.699270 13395 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } }
I20251210 14:58:37.699445 13395 raft_consensus.cc:385] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:37.699517 13395 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15073e88db734900ac12653cfa564e88, State: Initialized, Role: FOLLOWER
I20251210 14:58:37.699558 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:37.688515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.6.115.126:45729)
1210 14:58:37.688971 (+ 456us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:37.688980 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:37.689012 (+ 32us) server_negotiation.cc:408] Connection header received
1210 14:58:37.689568 (+ 556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:37.689586 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:37.689861 (+ 275us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:37.690175 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:37.691082 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.691881 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:37.692531 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.692795 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:37.695480 (+ 2685us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:37.695495 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:37.695505 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:37.695534 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:37.697253 (+ 1719us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:37.697685 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:37.697690 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:37.697694 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:37.697759 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:37.697971 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:37.697975 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:37.697977 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:37.698270 (+ 293us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:37.698474 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:37.698743 (+ 269us) server_negotiation.cc:300] Negotiation successful
1210 14:58:37.698972 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20251210 14:58:37.699966 13395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } }
I20251210 14:58:37.700093 13395 raft_consensus.cc:399] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:37.700166 13395 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:37.700254 13395 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:37.701006 13395 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } }
I20251210 14:58:37.701324 13395 leader_election.cc:304] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15073e88db734900ac12653cfa564e88; no voters:
I20251210 14:58:37.701642 13395 leader_election.cc:290] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:37.701995 13400 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:37.702173 13400 raft_consensus.cc:697] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 LEADER]: Becoming Leader. State: Replica: 15073e88db734900ac12653cfa564e88, State: Running, Role: LEADER
I20251210 14:58:37.702447 13400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } }
I20251210 14:58:37.702632 13395 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:37.703857 13402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15073e88db734900ac12653cfa564e88. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } }
I20251210 14:58:37.703965 13402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:37.704241 13401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } }
I20251210 14:58:37.704339 13401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:37.704319 13409 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:37.706954 13409 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:37.711946 13409 catalog_manager.cc:1357] Generated new cluster ID: 2a088d2cbc5f4ff9b8e0f24e0e789f20
I20251210 14:58:37.712015 13409 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:37.725210 13409 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:37.726354 13409 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:37.732692 13409 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Generated new TSK 0
I20251210 14:58:37.733444 13409 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:37.774657 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45729
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:37.877147 13423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:37.877384 13423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:37.877449 13423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:37.880708 13423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:37.880769 13423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:37.880892 13423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:37.885063 13423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:37.886092 13423 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:37.886926 13423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:37.893211 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:37.893216 13431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:37.893195 13428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:37.893400 13423 server_base.cc:1047] running on GCE node
I20251210 14:58:37.893831 13423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:37.894351 13423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:37.895510 13423 hybrid_clock.cc:648] HybridClock initialized: now 1765378717895495 us; error 28 us; skew 500 ppm
Dec 10 14:58:37 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:37.897975 13423 init.cc:377] Logged in from keytab as oryx/127.6.115.65@KRBTEST.COM (short username oryx)
I20251210 14:58:37.898972 13423 webserver.cc:492] Webserver started at http://127.6.115.65:42167/ using document root <none> and password file <none>
I20251210 14:58:37.899498 13423 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:37.899550 13423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:37.899732 13423 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:37.901315 13423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "0f17f9a97d6842679ebee2dfca09fd81"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "2602b621fddab2db052c2f8ff1fd5d4f"
server_key_iv: "c46f101785e0dc8c14e021c5958b09aa"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.901794 13423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "0f17f9a97d6842679ebee2dfca09fd81"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "2602b621fddab2db052c2f8ff1fd5d4f"
server_key_iv: "c46f101785e0dc8c14e021c5958b09aa"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.905028 13423 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251210 14:58:37.907136 13438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:37.908188 13423 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251210 14:58:37.908298 13423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "0f17f9a97d6842679ebee2dfca09fd81"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "2602b621fddab2db052c2f8ff1fd5d4f"
server_key_iv: "c46f101785e0dc8c14e021c5958b09aa"
server_key_version: "encryptionkey@0"
I20251210 14:58:37.908406 13423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:37.926807 13423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:37.929463 13423 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:37.929626 13423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:37.930114 13423 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:37.930881 13423 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:37.930932 13423 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:37.930996 13423 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:37.931043 13423 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:37.940541 13423 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:38623
I20251210 14:58:37.940575 13551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:38623 every 8 connection(s)
I20251210 14:58:37.941545 13423 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:37.949842 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13423
I20251210 14:58:37.949971 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:37.950245 6605 external_mini_cluster.cc:1468] Setting key 0c289c0bd7f098f12f0605a5dbd77765
Dec 10 14:58:37 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:37.954670 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:37.943451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:54469 (local address 127.6.115.126:45729)
1210 14:58:37.943726 (+ 275us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:37.943731 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:37.944355 (+ 624us) server_negotiation.cc:408] Connection header received
1210 14:58:37.945630 (+ 1275us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:37.945635 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:37.945698 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:37.945799 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:37.947699 (+ 1900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.948220 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:37.948804 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.948940 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:37.951378 (+ 2438us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:37.951397 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:37.951400 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:37.951422 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:37.952842 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:37.953299 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:37.953321 (+ 22us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:37.953323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:37.953363 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:37.953664 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:37.953667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:37.953669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:37.953834 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:37.953928 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:37.954383 (+ 455us) server_negotiation.cc:300] Negotiation successful
1210 14:58:37.954516 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251210 14:58:37.955268 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:37.943669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:54469)
1210 14:58:37.944205 (+ 536us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:37.944252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:37.945395 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:37.945984 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:37.945997 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:37.946554 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:37.947545 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:37.947559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.948338 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:37.948340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:37.948710 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:37.948716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:37.948870 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:37.949543 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:37.949566 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:37.951226 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:37.952949 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:37.952954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:37.952965 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:37.953204 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:37.953472 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:37.953474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:37.953476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:37.953560 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:37.953938 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:37.953946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:37.954155 (+ 209us) client_negotiation.cc:769] Sending connection context
1210 14:58:37.954346 (+ 191us) client_negotiation.cc:240] Negotiation successful
1210 14:58:37.954593 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20251210 14:58:37.956440 13552 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:37.956686 13552 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:37.957185 13552 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
I20251210 14:58:37.958693 13359 ts_manager.cc:194] Registered new tserver with Master: 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
WARNING: no policy specified for oryx/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:37.959885 13359 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:54469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:37.993885 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45729
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:38.098706 13559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:38.098934 13559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:38.098989 13559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:38.102209 13559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:38.102268 13559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:38.102376 13559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:38.106575 13559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:38.107527 13559 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:38.108381 13559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:38.114743 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.114751 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.114934 13559 server_base.cc:1047] running on GCE node
W20251210 14:58:38.114746 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.115523 13559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:38.116087 13559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:38.117254 13559 hybrid_clock.cc:648] HybridClock initialized: now 1765378718117245 us; error 26 us; skew 500 ppm
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:38.119815 13559 init.cc:377] Logged in from keytab as oryx/127.6.115.66@KRBTEST.COM (short username oryx)
I20251210 14:58:38.120813 13559 webserver.cc:492] Webserver started at http://127.6.115.66:33477/ using document root <none> and password file <none>
I20251210 14:58:38.121400 13559 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.121466 13559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.121665 13559 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:38.123325 13559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "e554247114c24a86a457a3600f421282"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "39bb641a52fba6719c05b282bb9aa62c"
server_key_iv: "b5517e7bce4d569e529c1ac6f80a2efd"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.123796 13559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "e554247114c24a86a457a3600f421282"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "39bb641a52fba6719c05b282bb9aa62c"
server_key_iv: "b5517e7bce4d569e529c1ac6f80a2efd"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.127173 13559 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:38.129146 13574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.130162 13559 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:38.130291 13559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "e554247114c24a86a457a3600f421282"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "39bb641a52fba6719c05b282bb9aa62c"
server_key_iv: "b5517e7bce4d569e529c1ac6f80a2efd"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.130379 13559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.144085 13559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:38.146842 13559 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:38.147017 13559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:38.147514 13559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:38.148332 13559 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:38.148378 13559 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.148435 13559 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:38.148461 13559 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.156870 13559 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:45829
I20251210 14:58:38.156899 13687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:45829 every 8 connection(s)
I20251210 14:58:38.157799 13559 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:38.158991 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13559
I20251210 14:58:38.159157 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:38.159420 6605 external_mini_cluster.cc:1468] Setting key 13914e3078d18c5bb62f98a891b08c06
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.67@KRBTEST.COM" created.
I20251210 14:58:38.171144 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.159484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40241 (local address 127.6.115.126:45729)
1210 14:58:38.159619 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.159622 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.160349 (+ 727us) server_negotiation.cc:408] Connection header received
1210 14:58:38.161168 (+ 819us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.161172 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.161225 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.161330 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.163366 (+ 2036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.163884 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.164780 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.164922 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.167479 (+ 2557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.167495 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.167497 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.167520 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.169148 (+ 1628us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.169706 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.169710 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.169713 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.169762 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.170067 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.170070 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.170071 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.170217 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.170315 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.170809 (+ 494us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.170933 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:58:38.171705 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.159793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:40241)
1210 14:58:38.160221 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.160255 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.160985 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.161513 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.161521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.161881 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.163190 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.163210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.164009 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.164016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.164659 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.164671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.164894 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.165630 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.165668 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.167334 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.169287 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.169293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.169304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.169605 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.169852 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.169858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.169860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.169957 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.170338 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.170345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.170555 (+ 210us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.170746 (+ 191us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.170998 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":86,"threads_started":1}
I20251210 14:58:38.172866 13688 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:38.173115 13688 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:38.173744 13688 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
I20251210 14:58:38.174865 13359 ts_manager.cc:194] Registered new tserver with Master: e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:38.175386 13359 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:40241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:38.206606 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:45729
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:38.308012 13695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:38.308197 13695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:38.308247 13695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:38.311342 13695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:38.311396 13695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:38.311470 13695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:38.315477 13695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:38.316401 13695 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:38.317180 13695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:38.323206 13700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.323180 13701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.323176 13703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.323410 13695 server_base.cc:1047] running on GCE node
I20251210 14:58:38.323812 13695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:38.324332 13695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:38.325434 13695 hybrid_clock.cc:648] HybridClock initialized: now 1765378718325419 us; error 31 us; skew 500 ppm
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:38.328032 13695 init.cc:377] Logged in from keytab as oryx/127.6.115.67@KRBTEST.COM (short username oryx)
I20251210 14:58:38.328928 13695 webserver.cc:492] Webserver started at http://127.6.115.67:33935/ using document root <none> and password file <none>
I20251210 14:58:38.329469 13695 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.329535 13695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.329697 13695 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:38.331362 13695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "27193df16bea4b3689ad1f635c99d1f4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "a0025272fcd596152febb9264ce443c8"
server_key_iv: "f51671bcfc9ece490962f6e31e0e6190"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.331830 13695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "27193df16bea4b3689ad1f635c99d1f4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "a0025272fcd596152febb9264ce443c8"
server_key_iv: "f51671bcfc9ece490962f6e31e0e6190"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.335039 13695 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:38.337085 13710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.337994 13695 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:38.338096 13695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "27193df16bea4b3689ad1f635c99d1f4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "a0025272fcd596152febb9264ce443c8"
server_key_iv: "f51671bcfc9ece490962f6e31e0e6190"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.338171 13695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.378742 13695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:38.381829 13695 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:38.382031 13695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:38.382598 13695 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:38.383494 13695 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:38.383550 13695 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.383625 13695 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:38.383658 13695 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.393221 13695 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:43469
I20251210 14:58:38.393289 13823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:43469 every 8 connection(s)
I20251210 14:58:38.394176 13695 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:38.401723 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13695
I20251210 14:58:38.401837 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:38.402149 6605 external_mini_cluster.cc:1468] Setting key 8a287858d6ffbc3f05c1930c66ce69e2
I20251210 14:58:38.406280 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.395840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:42795 (local address 127.6.115.126:45729)
1210 14:58:38.395985 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.395989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.396605 (+ 616us) server_negotiation.cc:408] Connection header received
1210 14:58:38.397465 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.397468 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.397526 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.397604 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.399192 (+ 1588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.399656 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.400262 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.400442 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.402582 (+ 2140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.402601 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.402604 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.402631 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.404523 (+ 1892us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.404945 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.404949 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.404951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.404999 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.405259 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.405263 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.405266 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.405468 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.405556 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.406018 (+ 462us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.406132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251210 14:58:38.406919 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.396073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:42795)
1210 14:58:38.396483 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.396516 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.397211 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.397759 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.397767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.398120 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.399051 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.399063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.399758 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.399761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.400154 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.400161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.400364 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.400943 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.400964 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.402446 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.404627 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.404632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.404642 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.404852 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.405077 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.405082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.405084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.405172 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.405576 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.405582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.405775 (+ 193us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.405959 (+ 184us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.406199 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20251210 14:58:38.407986 13824 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:38.408182 13824 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:38.408597 13824 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
I20251210 14:58:38.409466 13359 ts_manager.cc:194] Registered new tserver with Master: 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67:43469)
I20251210 14:58:38.409917 13359 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:42795
I20251210 14:58:38.416090 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.115.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:38.458078 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.125:35771
--webserver_interface=127.6.115.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:45729,127.6.115.125:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:38.557859 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:38.558051 13831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:38.558106 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:38.561218 13831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:38.561273 13831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:38.561295 13831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:38.561312 13831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:38.561333 13831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:38.565299 13831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:45729,127.6.115.125:35771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.125:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.115.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
W20251210 14:58:38.565469 13831 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.115.126:45729,127.6.115.125:35771'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251210 14:58:38.566212 13831 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:38.566991 13831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:38.572186 13836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.572175 13837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.572175 13839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.572435 13831 server_base.cc:1047] running on GCE node
I20251210 14:58:38.572877 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:38.573753 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:38.574894 13831 hybrid_clock.cc:648] HybridClock initialized: now 1765378718574884 us; error 25 us; skew 500 ppm
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:38.577497 13831 init.cc:377] Logged in from keytab as oryx/127.6.115.125@KRBTEST.COM (short username oryx)
I20251210 14:58:38.578433 13831 webserver.cc:492] Webserver started at http://127.6.115.125:34085/ using document root <none> and password file <none>
I20251210 14:58:38.578912 13831 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.578963 13831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.579128 13831 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:38.580590 13831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/instance:
uuid: "b70b290d8d4e444fae0462d65682bfa4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "72685aeda2a7b13e4d35960caef44ccd"
server_key_iv: "be69b12822de94e58bb4d98b884f1f1d"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.581000 13831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal/instance:
uuid: "b70b290d8d4e444fae0462d65682bfa4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "72685aeda2a7b13e4d35960caef44ccd"
server_key_iv: "be69b12822de94e58bb4d98b884f1f1d"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.583971 13831 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:58:38.586019 13846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.586910 13831 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:38.587008 13831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
uuid: "b70b290d8d4e444fae0462d65682bfa4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "72685aeda2a7b13e4d35960caef44ccd"
server_key_iv: "be69b12822de94e58bb4d98b884f1f1d"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.587088 13831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.595536 13831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:38.598119 13831 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:38.598294 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:38.605304 13831 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.125:35771
I20251210 14:58:38.605324 13898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.125:35771 every 8 connection(s)
I20251210 14:58:38.606181 13831 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/info.pb
I20251210 14:58:38.608669 13899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:38.612102 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } has no permanent_uuid. Determining permanent_uuid...
I20251210 14:58:38.612627 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13831
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.125@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:38.624537 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.613781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.6.115.126:45729)
1210 14:58:38.613901 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.613906 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.614459 (+ 553us) server_negotiation.cc:408] Connection header received
1210 14:58:38.615019 (+ 560us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.615023 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.615086 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.615164 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.616357 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.617231 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.618068 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.618207 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.620681 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.620699 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.620701 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.620724 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.622530 (+ 1806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.623056 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.623063 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.623066 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.623119 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.623423 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.623425 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.623426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.623595 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.623694 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.624261 (+ 567us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.624389 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:58:38.625165 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.613862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.6.115.125:35771)
1210 14:58:38.614233 (+ 371us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.614240 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.614268 (+ 28us) server_negotiation.cc:408] Connection header received
1210 14:58:38.614789 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.614805 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.614976 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.615130 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.616342 (+ 1212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.617449 (+ 1107us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.618290 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.618455 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.621051 (+ 2596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.621067 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.621080 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.621126 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.623190 (+ 2064us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.623628 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.623636 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.623640 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.623694 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.623933 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.623935 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.623936 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.624105 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.624336 (+ 231us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.624532 (+ 196us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.624667 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20251210 14:58:38.625169 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.613862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35300)
1210 14:58:38.614308 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.614342 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.614823 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.615257 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.615265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.615770 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.616187 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.616216 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.617402 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.617409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.617940 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.617951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.618153 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.618970 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.619008 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.620555 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.622651 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.622658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.622664 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.622949 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.623215 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.623217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.623219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.623326 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.623700 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.623705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.623966 (+ 261us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.624204 (+ 238us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.624481 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"spinlock_wait_cycles":54528,"thread_start_us":143,"threads_started":1}
I20251210 14:58:38.627158 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } has no permanent_uuid. Determining permanent_uuid...
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.125@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:38.633708 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.627542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55760)
1210 14:58:38.627651 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.627663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.627720 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.627897 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.627899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.628033 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.628184 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.628189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.628860 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.628862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.629189 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.629195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.629286 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.629741 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.629752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.631093 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.632764 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.632767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.632768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.632910 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.633080 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.633082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.633083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.633118 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.633346 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.633348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.633448 (+ 100us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.633518 (+ 70us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.633598 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20251210 14:58:38.633781 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.627632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.6.115.125:35771)
1210 14:58:38.627728 (+ 96us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.627731 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.627763 (+ 32us) server_negotiation.cc:408] Connection header received
1210 14:58:38.627787 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.627789 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.627823 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.627880 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.628293 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.628770 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.629280 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.629465 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.631228 (+ 1763us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.631243 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.631246 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.631273 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.632673 (+ 1400us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.632984 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.632986 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.632987 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.633019 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.633175 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.633177 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.633178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.633288 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.633349 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.633549 (+ 200us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.633645 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":38}
I20251210 14:58:38.636056 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Bootstrap starting.
I20251210 14:58:38.638307 13899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:38.638978 13899 log.cc:826] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:38.640476 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: No bootstrap required, opened a new log
I20251210 14:58:38.642836 13899 raft_consensus.cc:359] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } }
I20251210 14:58:38.643002 13899 raft_consensus.cc:385] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:38.643038 13899 raft_consensus.cc:740] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70b290d8d4e444fae0462d65682bfa4, State: Initialized, Role: FOLLOWER
I20251210 14:58:38.643440 13899 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } }
I20251210 14:58:38.643846 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } }
I20251210 14:58:38.643963 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:38.644430 13899 sys_catalog.cc:565] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: configured and running, proceeding with master startup.
W20251210 14:58:38.648773 13916 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251210 14:58:38.648845 13916 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251210 14:58:38.650480 13831 master_runner.cc:428] Detected that this master b70b290d8d4e444fae0462d65682bfa4 is joining an existing cluster
I20251210 14:58:38.650540 13831 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.115.125:35771
I20251210 14:58:38.659350 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.652772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.6.115.125:35771)
1210 14:58:38.652902 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.652906 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.653307 (+ 401us) server_negotiation.cc:408] Connection header received
1210 14:58:38.653446 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.653449 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.653490 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.653568 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.654221 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.654670 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.655215 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.655419 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.656252 (+ 833us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.656267 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.656270 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.656292 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.658032 (+ 1740us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.658383 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.658386 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.658388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.658433 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.658683 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.658686 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.658688 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.658814 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.658897 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.659091 (+ 194us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.659233 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":68}
I20251210 14:58:38.659377 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.652642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55770)
1210 14:58:38.653221 (+ 579us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.653233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.653319 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.653594 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.653597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.653802 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.654095 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.654103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.654775 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.654778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.655107 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.655113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.655221 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.655849 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.655861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.656120 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.658136 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.658139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.658140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.658295 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.658536 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.658538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.658539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.658584 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.658894 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.658897 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.658961 (+ 64us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.659047 (+ 86us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.659165 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":279,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.6.115.124@KRBTEST.COM; defaulting to no policy
I20251210 14:58:38.686092 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.652682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.6.115.126:45729)
1210 14:58:38.652820 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.652824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.653022 (+ 198us) server_negotiation.cc:408] Connection header received
1210 14:58:38.653216 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.653219 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.653261 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.653316 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.654372 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.655113 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.656032 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.656236 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.682576 (+ 26340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.682594 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.682597 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.682625 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.684474 (+ 1849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.684930 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.684934 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.684936 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.684977 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.685220 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.685223 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.685225 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.685443 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.685515 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.685843 (+ 328us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.685954 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:58:38.686566 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.652642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35304)
1210 14:58:38.652923 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.652942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.653053 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.653482 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.653485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.653726 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.654250 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.654258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.655273 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.655277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.655895 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.655905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.656342 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.682037 (+ 25695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.682054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.682421 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.684603 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.684607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.684609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.684827 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.685082 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.685085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.685087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.685131 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.685583 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.685586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.685684 (+ 98us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.686275 (+ 591us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.686403 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.124@KRBTEST.COM" created.
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.126@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:38.700448 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.691316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.6.115.125:35771)
1210 14:58:38.691419 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.691423 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.691581 (+ 158us) server_negotiation.cc:408] Connection header received
1210 14:58:38.691757 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.691760 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.691806 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.691880 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.692839 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.693513 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.694330 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.694511 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.696909 (+ 2398us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.696924 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.696927 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.696949 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.698706 (+ 1757us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.699189 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.699192 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.699194 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.699234 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.699486 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.699488 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.699490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.699631 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.699691 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.700146 (+ 455us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.700272 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251210 14:58:38.700523 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.690680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55776)
1210 14:58:38.691479 (+ 799us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.691513 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.691646 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.691995 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.692001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.692425 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.692715 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.692721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.693643 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.693657 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.694193 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.694201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.694606 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.695148 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.695163 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.696778 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.698823 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.698830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.698835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.699085 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.699355 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.699358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.699360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.699407 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.699762 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.699768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.699866 (+ 98us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.700071 (+ 205us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.700230 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":58,"threads_started":1}
I20251210 14:58:38.701855 13359 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.6.115.125:35771: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4"
member_type: NON_VOTER
last_known_addr {
host: "127.6.115.125"
port: 35771
}
attrs {
promote: true
}
}
dest_uuid: "15073e88db734900ac12653cfa564e88"
cas_config_opid_index: -1
I20251210 14:58:38.702483 13359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } }
I20251210 14:58:38.704114 13927 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:38.704871 13873 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:38.706250 13873 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251210 14:58:38.706786 13928 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:38.707335 13927 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.6.115.125:35771
I20251210 14:58:38.707749 13831 master.cc:561] Master@127.6.115.125:35771 shutting down...
I20251210 14:58:38.707865 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:38.707947 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
W20251210 14:58:38.709649 13343 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251210 14:58:38.711532 13343 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 -> Peer b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125:35771): Couldn't send request to peer b70b290d8d4e444fae0462d65682bfa4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251210 14:58:38.712313 13831 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:38.712589 13831 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:38.712709 13831 tablet_replica.cc:333] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: stopping tablet replica
WARNING: no policy specified for HTTP/127.6.115.124@KRBTEST.COM; defaulting to no policy
I20251210 14:58:38.717020 13831 master.cc:583] Master@127.6.115.125:35771 shutdown complete.
I20251210 14:58:38.717070 13831 master_runner.cc:305] Clearing existing system tablet
I20251210 14:58:38.717271 13831 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.717319 13831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.718436 13831 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.124@KRBTEST.COM" created.
I20251210 14:58:38.719200 13935 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.720767 13831 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.720885 13831 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:38.720937 13831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
uuid: "b70b290d8d4e444fae0462d65682bfa4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "72685aeda2a7b13e4d35960caef44ccd"
server_key_iv: "be69b12822de94e58bb4d98b884f1f1d"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.722790 13831 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:38.724892 13831 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251210 14:58:38.724975 13831 log.cc:1199] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251210 14:58:38.725214 13831 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Deleting consensus metadata
I20251210 14:58:38.725425 13831 master_runner.cc:315] Copying system tablet from 127.6.115.126:45729
I20251210 14:58:38.726502 13831 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.115.126:45729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:38.733958 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.124:41865
--webserver_interface=127.6.115.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251210 14:58:38.735839 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.727067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.6.115.126:45729)
1210 14:58:38.727218 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.727225 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.727275 (+ 50us) server_negotiation.cc:408] Connection header received
1210 14:58:38.727410 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.727412 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.727456 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.727542 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.728214 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.728777 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.729628 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.729748 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.730763 (+ 1015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.730778 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.730780 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.730802 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.734133 (+ 3331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.734632 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.734653 (+ 21us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.734670 (+ 17us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.734725 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.734999 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.735005 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.735010 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.735226 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.735335 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.735611 (+ 276us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.735711 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:58:38.735847 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.726940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35310)
1210 14:58:38.727184 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.727205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.727322 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.727576 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.727579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.727796 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.728055 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.728062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.728895 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.728900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.729520 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.729541 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.729640 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.730275 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.730291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.730659 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.734304 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.734307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.734310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.734526 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.734809 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.734813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.734815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.734878 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.735376 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.735379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.735454 (+ 75us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.735546 (+ 92us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.735677 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20251210 14:58:38.736497 13379 tablet_copy_service.cc:140] P 15073e88db734900ac12653cfa564e88: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b70b290d8d4e444fae0462d65682bfa4 ({username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:35310)
I20251210 14:58:38.736654 13379 tablet_copy_service.cc:161] P 15073e88db734900ac12653cfa564e88: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b70b290d8d4e444fae0462d65682bfa4 at {username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:35310: session id = b70b290d8d4e444fae0462d65682bfa4-00000000000000000000000000000000
I20251210 14:58:38.738183 13379 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Tablet Copy: opened 0 blocks and 1 log segments
I20251210 14:58:38.739758 13831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251210 14:58:38.742272 13831 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: tablet copy: Starting download of 0 data blocks...
I20251210 14:58:38.742448 13831 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: tablet copy: Starting download of 1 WAL segments...
I20251210 14:58:38.744344 13831 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251210 14:58:38.746050 13379 tablet_copy_service.cc:342] P 15073e88db734900ac12653cfa564e88: Request end of tablet copy session b70b290d8d4e444fae0462d65682bfa4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:35310
I20251210 14:58:38.746214 13379 tablet_copy_service.cc:434] P 15073e88db734900ac12653cfa564e88: ending tablet copy session b70b290d8d4e444fae0462d65682bfa4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b70b290d8d4e444fae0462d65682bfa4
W20251210 14:58:38.751452 13831 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251210 14:58:38.753625 13831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:38.755220 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.755285 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.755357 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.756712 13831 server_base.cc:1047] running on GCE node
I20251210 14:58:38.756910 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:38.757185 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:38.758381 13831 hybrid_clock.cc:648] HybridClock initialized: now 1765378718758340 us; error 61 us; skew 500 ppm
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:38.760057 13831 init.cc:377] Logged in from keytab as oryx/127.6.115.125@KRBTEST.COM (short username oryx)
I20251210 14:58:38.760524 13831 webserver.cc:492] Webserver started at http://127.6.115.125:41613/ using document root <none> and password file <none>
I20251210 14:58:38.760703 13831 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.760743 13831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.762084 13831 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:38.763089 13954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.763394 13831 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:38.763468 13831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
uuid: "b70b290d8d4e444fae0462d65682bfa4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "72685aeda2a7b13e4d35960caef44ccd"
server_key_iv: "be69b12822de94e58bb4d98b884f1f1d"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.763541 13831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.775760 13831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:38.777835 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:38.782251 13831 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.125:35771
I20251210 14:58:38.782832 13831 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-1/data/info.pb
I20251210 14:58:38.783232 14006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.125:35771 every 8 connection(s)
I20251210 14:58:38.784475 14007 sys_catalog.cc:263] Verifying existing consensus state
I20251210 14:58:38.785354 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Bootstrap starting.
I20251210 14:58:38.794790 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251210 14:58:38.795063 14007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Bootstrap complete.
I20251210 14:58:38.795688 14007 raft_consensus.cc:359] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } }
I20251210 14:58:38.795814 14007 raft_consensus.cc:740] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b70b290d8d4e444fae0462d65682bfa4, State: Initialized, Role: LEARNER
I20251210 14:58:38.795923 14007 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } }
I20251210 14:58:38.796236 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:38.796409 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: LEARNER
I20251210 14:58:38.796419 14007 sys_catalog.cc:565] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:38.799867 14021 catalog_manager.cc:1269] Loaded cluster ID: 2a088d2cbc5f4ff9b8e0f24e0e789f20
I20251210 14:58:38.799917 14021 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: loading cluster ID for follower catalog manager: success
I20251210 14:58:38.801983 14021 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: acquiring CA information for follower catalog manager: success
I20251210 14:58:38.803334 14021 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251210 14:58:38.851336 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:38.851516 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:38.851565 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:38.854756 13943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:38.854808 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:38.854830 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:38.854849 13943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:38.854866 13943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:38.858949 13943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.124:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.115.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:38.859822 13943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:38.860575 13943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251210 14:58:38.865857 13943 server_base.cc:1047] running on GCE node
W20251210 14:58:38.865818 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.865836 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:38.865799 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:38.866520 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:38.867298 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:38.868432 13943 hybrid_clock.cc:648] HybridClock initialized: now 1765378718868418 us; error 30 us; skew 500 ppm
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:38.870903 13943 init.cc:377] Logged in from keytab as oryx/127.6.115.124@KRBTEST.COM (short username oryx)
I20251210 14:58:38.871780 13943 webserver.cc:492] Webserver started at http://127.6.115.124:45621/ using document root <none> and password file <none>
I20251210 14:58:38.872272 13943 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:38.872316 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:38.872457 13943 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:38.874027 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/instance:
uuid: "e37585beb7a84c379041754b1aceb780"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "7dc7978c1adc79a393014a2877d478aa"
server_key_iv: "783598b1a98f88c857addd49ec53a85e"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.874441 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal/instance:
uuid: "e37585beb7a84c379041754b1aceb780"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "7dc7978c1adc79a393014a2877d478aa"
server_key_iv: "783598b1a98f88c857addd49ec53a85e"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.877476 13943 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:38.879537 14036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:38.880511 13943 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:38.880601 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
uuid: "e37585beb7a84c379041754b1aceb780"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "7dc7978c1adc79a393014a2877d478aa"
server_key_iv: "783598b1a98f88c857addd49ec53a85e"
server_key_version: "encryptionkey@0"
I20251210 14:58:38.880692 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:38.892545 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:38.895239 13943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:38.895395 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:38.902390 13943 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.124:41865
I20251210 14:58:38.902401 14088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.124:41865 every 8 connection(s)
I20251210 14:58:38.903340 13943 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/info.pb
I20251210 14:58:38.906010 14089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:38.909571 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 13943
I20251210 14:58:38.909770 14089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } has no permanent_uuid. Determining permanent_uuid...
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.124@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:38.920863 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.912124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.6.115.126:45729)
1210 14:58:38.912271 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.912275 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.912473 (+ 198us) server_negotiation.cc:408] Connection header received
1210 14:58:38.912619 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.912622 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.912680 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.912759 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.913932 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.914445 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.915072 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.915277 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.917884 (+ 2607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.917904 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.917906 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.917930 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.919321 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.919706 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.919709 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.919710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.919768 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.920024 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.920027 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.920028 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.920157 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.920238 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.920624 (+ 386us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.920722 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251210 14:58:38.921554 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.911906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35314)
1210 14:58:38.912356 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.912392 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.912524 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.912810 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.912820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.913175 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.913713 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.913723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.914577 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.914583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.914964 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.914971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.915495 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.915946 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.915972 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.917753 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.919425 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.919431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.919436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.919610 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.919851 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.919854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.919855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.919913 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.920259 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.920265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.920387 (+ 122us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.920568 (+ 181us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.920964 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":190,"spinlock_wait_cycles":1408,"thread_start_us":117,"threads_started":1}
I20251210 14:58:38.921554 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.910692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.6.115.124:41865)
1210 14:58:38.911071 (+ 379us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.911079 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.911111 (+ 32us) server_negotiation.cc:408] Connection header received
1210 14:58:38.911672 (+ 561us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.911693 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.912039 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.913205 (+ 1166us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.913536 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.914260 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.914950 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.915197 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.917348 (+ 2151us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.917385 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.917396 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.917421 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.919261 (+ 1840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.919646 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.919653 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.919657 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.919725 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.919971 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.919977 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.919979 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.920285 (+ 306us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.920461 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.920605 (+ 144us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.920796 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20251210 14:58:38.922670 14089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } has no permanent_uuid. Determining permanent_uuid...
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.124@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:38.929754 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.923097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55786)
1210 14:58:38.923232 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.923245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.923318 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.923711 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.923713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.923837 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.924017 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.924022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.924902 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.924904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.925283 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.925290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.925421 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.925833 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.925845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.926896 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.928606 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.928609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.928610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.928768 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.929017 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.929019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.929021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.929063 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.929343 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.929345 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.929451 (+ 106us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.929537 (+ 86us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.929633 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20251210 14:58:38.929833 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.923229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.6.115.125:35771)
1210 14:58:38.923494 (+ 265us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.923497 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.923509 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:38.923541 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.923543 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.923580 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.923652 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.924132 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.924796 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.925432 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.925582 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.927014 (+ 1432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.927029 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.927031 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.927053 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.928494 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.928877 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.928880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.928882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.928931 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.929135 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.929137 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.929139 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.929273 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.929348 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.929588 (+ 240us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.929699 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20251210 14:58:38.930444 14089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
Dec 10 14:58:38 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378718, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.124@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
I20251210 14:58:38.936559 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.930833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53124)
1210 14:58:38.930961 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.930974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.931041 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.931219 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.931222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.931346 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.931457 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.931461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.932016 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.932019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.932291 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.932296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.932381 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.932799 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.932809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.933919 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.935480 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.935483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.935485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.935630 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.935824 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.935828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.935831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.935870 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.936139 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.936142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.936211 (+ 69us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.936295 (+ 84us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.936408 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20251210 14:58:38.936578 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.930882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.6.115.124:41865)
1210 14:58:38.931020 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.931024 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.931034 (+ 10us) server_negotiation.cc:408] Connection header received
1210 14:58:38.931108 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.931112 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.931142 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.931202 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.931549 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.931930 (+ 381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.932373 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.932554 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.934025 (+ 1471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.934047 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.934050 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.934075 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.935380 (+ 1305us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.935712 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.935715 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.935718 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.935751 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.935945 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.935947 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.935949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.936058 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.936139 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.936340 (+ 201us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.936467 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":69}
I20251210 14:58:38.938997 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Bootstrap starting.
I20251210 14:58:38.941272 14089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:38.942020 14089 log.cc:826] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:38.943818 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: No bootstrap required, opened a new log
I20251210 14:58:38.946206 14089 raft_consensus.cc:359] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } }
I20251210 14:58:38.946353 14089 raft_consensus.cc:385] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:38.946386 14089 raft_consensus.cc:740] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37585beb7a84c379041754b1aceb780, State: Initialized, Role: FOLLOWER
I20251210 14:58:38.946745 14089 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } }
I20251210 14:58:38.947235 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } } }
I20251210 14:58:38.947363 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:38.947911 14089 sys_catalog.cc:565] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: configured and running, proceeding with master startup.
W20251210 14:58:38.952350 14106 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251210 14:58:38.952414 14106 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251210 14:58:38.956357 13943 master_runner.cc:428] Detected that this master e37585beb7a84c379041754b1aceb780 is joining an existing cluster
I20251210 14:58:38.956430 13943 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.115.124:41865
I20251210 14:58:38.962159 13552 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:38.978080 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.975334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35330)
1210 14:58:38.975458 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.975474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.975562 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.975913 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.975915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.975932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:38.976116 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.976121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.977159 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.977162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.977774 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.977781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.977864 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.977902 (+ 38us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.977937 (+ 35us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.977992 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20251210 14:58:38.978700 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.975397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.6.115.126:45729)
1210 14:58:38.975701 (+ 304us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.975706 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.975719 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:38.975769 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.975772 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.975818 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.975906 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:38.976218 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.977043 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.977869 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.978435 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.978494 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.978538 (+ 44us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.978590 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":11392,"thread_start_us":125,"threads_started":1}
I20251210 14:58:38.982445 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.979694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55810)
1210 14:58:38.979835 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.979848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.979911 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.980207 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.980210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.980225 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:38.980410 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.980416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.981087 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.981091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.982027 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.982038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.982149 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.982190 (+ 41us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.982242 (+ 52us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.982305 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20251210 14:58:38.982787 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.979782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.6.115.125:35771)
1210 14:58:38.980005 (+ 223us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.980008 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.980019 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:38.980055 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.980057 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.980100 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.980183 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:38.980498 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.980992 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.982134 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.982450 (+ 316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.982559 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.982613 (+ 54us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.982671 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20251210 14:58:38.996826 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.958424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53134)
1210 14:58:38.961770 (+ 3346us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.961779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.961846 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.962124 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.962127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.989739 (+ 27612us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.990066 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.990076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.990914 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.990917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.991424 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.991431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.991526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.992511 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.992531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.992923 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.995490 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.995494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.995496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.995696 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.995928 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.995931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.995933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.995995 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.996346 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.996349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.996425 (+ 76us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.996520 (+ 95us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.996637 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3304,"spinlock_wait_cycles":3968,"thread_start_us":118,"threads_started":1}
I20251210 14:58:38.997043 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.958398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:55794)
1210 14:58:38.961541 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.961552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.961624 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.961955 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.961982 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.990451 (+ 28469us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.990698 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.990705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.991657 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.991661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.992237 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.992245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.992364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.992874 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.992891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.993413 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.995692 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.995696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.995698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.995928 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.996170 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.996176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.996229 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.996618 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.996621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.996685 (+ 64us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.996764 (+ 79us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.996888 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3079,"spinlock_wait_cycles":1920,"thread_start_us":135,"threads_started":1}
I20251210 14:58:38.997403 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.959343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.6.115.126:45729)
1210 14:58:38.959473 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.959476 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.959487 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:38.959517 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.959519 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.959559 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.959620 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.990228 (+ 30608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.990937 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.992168 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.992360 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.993756 (+ 1396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.993775 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.993779 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.993808 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.995600 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996111 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.996114 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.996117 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.996171 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996420 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.996424 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.996425 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.996570 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.996648 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.997133 (+ 485us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.997243 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251210 14:58:38.997581 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.958816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.6.115.124:41865)
1210 14:58:38.958921 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.958924 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.961843 (+ 2919us) server_negotiation.cc:408] Connection header received
1210 14:58:38.961927 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.961931 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.961970 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.962037 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.990197 (+ 28160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.990812 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.991748 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.991919 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.993056 (+ 1137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.993070 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.993073 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.993094 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.995370 (+ 2276us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.995786 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.995793 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.995797 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.995839 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996079 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.996081 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.996083 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.996235 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.996316 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.997150 (+ 834us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.997271 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251210 14:58:38.997783 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.958364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35328)
1210 14:58:38.959105 (+ 741us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:38.959117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:38.959190 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:38.959681 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:38.959686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:38.989768 (+ 30082us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:38.990075 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.990084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.991404 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.991408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:38.992026 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:38.992038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.992192 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.993241 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:38.993261 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:38.993635 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:38.995725 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.995729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.995732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.995993 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.996264 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:38.996270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:38.996319 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:38.996686 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:38.996689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:38.996762 (+ 73us) client_negotiation.cc:769] Sending connection context
1210 14:58:38.996873 (+ 111us) client_negotiation.cc:240] Negotiation successful
1210 14:58:38.996994 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"spinlock_wait_cycles":2432,"thread_start_us":618,"threads_started":1}
I20251210 14:58:38.999107 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:38.958816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.6.115.125:35771)
1210 14:58:38.959259 (+ 443us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:38.959264 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:38.961620 (+ 2356us) server_negotiation.cc:408] Connection header received
1210 14:58:38.961706 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:38.961710 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:38.961760 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:38.961844 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:38.990828 (+ 28984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.991536 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:38.992555 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:38.992745 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:38.993541 (+ 796us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:38.993557 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:38.993563 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:38.993591 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:38.995580 (+ 1989us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996027 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.996033 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.996036 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.996081 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:38.996330 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:38.996335 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:38.996338 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:38.996496 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:38.996576 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:38.998823 (+ 2247us) server_negotiation.cc:300] Negotiation successful
1210 14:58:38.998933 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
Dec 10 14:58:39 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378717, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.126@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
I20251210 14:58:39.007961 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.001157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.6.115.124:41865)
1210 14:58:39.001262 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:39.001265 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:39.001385 (+ 120us) server_negotiation.cc:408] Connection header received
1210 14:58:39.001529 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:39.001532 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:39.001572 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:39.001634 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:39.002160 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.002547 (+ 387us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.003297 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.003410 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.005194 (+ 1784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:39.005207 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:39.005209 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:39.005230 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:39.006723 (+ 1493us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:39.007087 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:39.007092 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:39.007094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:39.007130 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:39.007330 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:39.007333 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:39.007335 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:39.007448 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:39.007535 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:39.007750 (+ 215us) server_negotiation.cc:300] Negotiation successful
1210 14:58:39.007839 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251210 14:58:39.007968 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.001087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53144)
1210 14:58:39.001290 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:39.001304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:39.001425 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:39.001660 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:39.001663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:39.001837 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:39.002060 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.002068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.002632 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.002636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:39.003194 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.003203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.003351 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.003794 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:39.003809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:39.005077 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:39.006827 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:39.006831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:39.006833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:39.007002 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:39.007206 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:39.007210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:39.007212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:39.007256 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:39.007537 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:39.007542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:39.007614 (+ 72us) client_negotiation.cc:769] Sending connection context
1210 14:58:39.007698 (+ 84us) client_negotiation.cc:240] Negotiation successful
1210 14:58:39.007815 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20251210 14:58:39.008750 13358 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.6.115.124:41865: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e37585beb7a84c379041754b1aceb780"
member_type: NON_VOTER
last_known_addr {
host: "127.6.115.124"
port: 41865
}
attrs {
promote: true
}
}
dest_uuid: "15073e88db734900ac12653cfa564e88"
cas_config_opid_index: 5
I20251210 14:58:39.009326 13358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.010308 13930 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e37585beb7a84c379041754b1aceb780 (127.6.115.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.010607 13981 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 LEARNER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251210 14:58:39.011004 13928 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251210 14:58:39.011651 13930 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.6.115.124:41865
I20251210 14:58:39.011818 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.011894 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.012686 14063 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:39.012671 13981 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e37585beb7a84c379041754b1aceb780 (127.6.115.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.014037 14063 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251210 14:58:39.014400 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15073e88db734900ac12653cfa564e88. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.014504 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: LEARNER
I20251210 14:58:39.014503 13930 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:39.014587 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.014653 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: LEARNER
I20251210 14:58:39.015852 13930 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: attempting to promote NON_VOTER b70b290d8d4e444fae0462d65682bfa4 to VOTER
I20251210 14:58:39.016131 13943 master.cc:561] Master@127.6.115.124:41865 shutting down...
I20251210 14:58:39.016206 13930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.018738 13981 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 LEARNER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20251210 14:58:39.018915 13346 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 -> Peer e37585beb7a84c379041754b1aceb780 (127.6.115.124:41865): Couldn't send request to peer e37585beb7a84c379041754b1aceb780. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251210 14:58:39.019269 13930 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251210 14:58:39.020844 14063 raft_consensus.cc:872] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.020826 13928 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
W20251210 14:58:39.020915 14063 raft_consensus.cc:1579] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.022176 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.022274 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.022671 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.022749 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.022549 14063 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:39.023381 14119 mvcc.cc:204] Tried to move back new op lower bound from 7230991227829653504 to 7230991227709988864. Current Snapshot: MvccSnapshot[applied={T|T < 7230991227739275264}]
I20251210 14:58:39.023319 13981 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.024485 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15073e88db734900ac12653cfa564e88. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:39.024569 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20251210 14:58:39.024652 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:39.024712 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20251210 14:58:39.024778 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: NON_VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: true } } }
I20251210 14:58:39.024830 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20251210 14:58:39.024930 14118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.025013 14118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:39.027616 13943 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20251210 14:58:39.027834 13943 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20251210 14:58:39.027911 13943 tablet_replica.cc:333] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: stopping tablet replica
I20251210 14:58:39.037684 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.037447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53146)
1210 14:58:39.037564 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:39.037616 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251210 14:58:39.037922 13359 master.cc:679] Network error: unable to get registration information for peer e37585beb7a84c379041754b1aceb780: Client connection negotiation failed: client connection to 127.6.115.124:41865: connect: Connection refused (error 111)
I20251210 14:58:39.042332 13943 master.cc:583] Master@127.6.115.124:41865 shutdown complete.
I20251210 14:58:39.042387 13943 master_runner.cc:305] Clearing existing system tablet
I20251210 14:58:39.042637 13943 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:39.042691 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:39.043932 13943 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251210 14:58:39.044826 14127 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:39.045047 13943 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:39.045161 13943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:39.045217 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
uuid: "e37585beb7a84c379041754b1aceb780"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "7dc7978c1adc79a393014a2877d478aa"
server_key_iv: "783598b1a98f88c857addd49ec53a85e"
server_key_version: "encryptionkey@0"
I20251210 14:58:39.046590 13943 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:39.048290 13943 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251210 14:58:39.048364 13943 log.cc:1199] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251210 14:58:39.048621 13943 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Deleting consensus metadata
I20251210 14:58:39.048772 13943 master_runner.cc:315] Copying system tablet from 127.6.115.126:45729
I20251210 14:58:39.049860 13943 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: tablet copy: Beginning tablet copy session from remote peer at address 127.6.115.126:45729
I20251210 14:58:39.056929 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.050423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.6.115.126:45729)
1210 14:58:39.050551 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:39.050555 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:39.050653 (+ 98us) server_negotiation.cc:408] Connection header received
1210 14:58:39.050759 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:39.050763 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:39.050814 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:39.050909 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:39.051395 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.052178 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.052882 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.053095 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.053844 (+ 749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:39.053862 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:39.053866 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:39.053892 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:39.055535 (+ 1643us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:39.055940 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:39.055944 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:39.055946 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:39.055993 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:39.056197 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:39.056200 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:39.056202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:39.056353 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:39.056461 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:39.056701 (+ 240us) server_negotiation.cc:300] Negotiation successful
1210 14:58:39.056802 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:58:39.056951 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.050337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:35340)
1210 14:58:39.050574 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:39.050587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:39.050681 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:39.050941 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:39.050943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:39.051104 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:39.051291 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.051296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.052393 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.052396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:39.052781 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.052789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.052881 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.053442 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:39.053455 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:39.053731 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:39.055660 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:39.055663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:39.055664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:39.055838 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:39.056077 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:39.056079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:39.056080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:39.056122 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:39.056451 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:39.056453 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:39.056526 (+ 73us) client_negotiation.cc:769] Sending connection context
1210 14:58:39.056615 (+ 89us) client_negotiation.cc:240] Negotiation successful
1210 14:58:39.056750 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20251210 14:58:39.057427 13379 tablet_copy_service.cc:140] P 15073e88db734900ac12653cfa564e88: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e37585beb7a84c379041754b1aceb780 ({username='oryx', principal='oryx/127.6.115.124@KRBTEST.COM'} at 127.0.0.1:35340)
I20251210 14:58:39.057533 13379 tablet_copy_service.cc:161] P 15073e88db734900ac12653cfa564e88: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e37585beb7a84c379041754b1aceb780 at {username='oryx', principal='oryx/127.6.115.124@KRBTEST.COM'} at 127.0.0.1:35340: session id = e37585beb7a84c379041754b1aceb780-00000000000000000000000000000000
I20251210 14:58:39.058248 13379 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Tablet Copy: opened 0 blocks and 1 log segments
I20251210 14:58:39.059348 13943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251210 14:58:39.060945 13943 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: tablet copy: Starting download of 0 data blocks...
I20251210 14:58:39.061085 13943 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: tablet copy: Starting download of 1 WAL segments...
I20251210 14:58:39.062577 13943 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251210 14:58:39.063926 13379 tablet_copy_service.cc:342] P 15073e88db734900ac12653cfa564e88: Request end of tablet copy session e37585beb7a84c379041754b1aceb780-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.115.124@KRBTEST.COM'} at 127.0.0.1:35340
I20251210 14:58:39.064002 13379 tablet_copy_service.cc:434] P 15073e88db734900ac12653cfa564e88: ending tablet copy session e37585beb7a84c379041754b1aceb780-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e37585beb7a84c379041754b1aceb780
W20251210 14:58:39.067984 13943 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251210 14:58:39.070176 13943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:39.071692 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:39.071802 13943 server_base.cc:1047] running on GCE node
W20251210 14:58:39.071910 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:39.071898 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:39.072185 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:39.072399 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:39.073563 13943 hybrid_clock.cc:648] HybridClock initialized: now 1765378719073549 us; error 28 us; skew 500 ppm
I20251210 14:58:39.074167 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.073891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53162)
1210 14:58:39.074045 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:39.074093 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.124:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251210 14:58:39.074359 13359 master.cc:679] Network error: unable to get registration information for peer e37585beb7a84c379041754b1aceb780: Client connection negotiation failed: client connection to 127.6.115.124:41865: connect: Connection refused (error 111)
Dec 10 14:58:39 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378719, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:39.075320 13943 init.cc:377] Logged in from keytab as oryx/127.6.115.124@KRBTEST.COM (short username oryx)
I20251210 14:58:39.075732 13943 webserver.cc:492] Webserver started at http://127.6.115.124:38967/ using document root <none> and password file <none>
I20251210 14:58:39.075906 13943 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:39.075939 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:39.076879 13943 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:39.077723 14144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:39.077934 13943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251210 14:58:39.077986 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
uuid: "e37585beb7a84c379041754b1aceb780"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "7dc7978c1adc79a393014a2877d478aa"
server_key_iv: "783598b1a98f88c857addd49ec53a85e"
server_key_version: "encryptionkey@0"
I20251210 14:58:39.078047 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:39.091845 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:39.093839 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:39.098865 13943 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.124:41865
I20251210 14:58:39.098884 14196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.124:41865 every 8 connection(s)
I20251210 14:58:39.099418 13943 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/master-2/data/info.pb
I20251210 14:58:39.101142 14197 sys_catalog.cc:263] Verifying existing consensus state
I20251210 14:58:39.102118 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Bootstrap starting.
I20251210 14:58:39.110710 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251210 14:58:39.110989 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: Bootstrap complete.
I20251210 14:58:39.111588 14197 raft_consensus.cc:359] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.111718 14197 raft_consensus.cc:740] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e37585beb7a84c379041754b1aceb780, State: Initialized, Role: LEARNER
I20251210 14:58:39.111848 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } }
I20251210 14:58:39.112164 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.112249 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: LEARNER
I20251210 14:58:39.112299 14197 sys_catalog.cc:565] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:39.115700 14211 catalog_manager.cc:1269] Loaded cluster ID: 2a088d2cbc5f4ff9b8e0f24e0e789f20
I20251210 14:58:39.115749 14211 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: loading cluster ID for follower catalog manager: success
I20251210 14:58:39.117259 14211 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: acquiring CA information for follower catalog manager: success
I20251210 14:58:39.118376 14211 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251210 14:58:39.144800 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.142288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:53164)
1210 14:58:39.142424 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:39.142441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:39.142509 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:39.142893 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:39.142895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:39.142909 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:39.143089 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.143093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.143946 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.143948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:39.144514 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:39.144520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.144602 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.144619 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:58:39.144654 (+ 35us) client_negotiation.cc:240] Negotiation successful
1210 14:58:39.144699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20251210 14:58:39.145308 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:39.142394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.6.115.124:41865)
1210 14:58:39.142616 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:39.142620 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:39.142632 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:39.142717 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:39.142721 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:39.142796 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:39.142888 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:39.143174 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.143843 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:39.144599 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:39.144923 (+ 324us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:39.145048 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:39.145089 (+ 41us) server_negotiation.cc:300] Negotiation successful
1210 14:58:39.145143 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20251210 14:58:39.177855 13688 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:39.411692 13824 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:39.421028 14171 raft_consensus.cc:1217] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251210 14:58:39.421394 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15073e88db734900ac12653cfa564e88. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.421537 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: LEARNER
I20251210 14:58:39.845201 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: NON_VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: true } } }
I20251210 14:58:39.845355 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.846342 14213 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: attempting to promote NON_VOTER e37585beb7a84c379041754b1aceb780 to VOTER
I20251210 14:58:39.846756 14213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:39.847935 14171 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 LEARNER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251210 14:58:39.847980 13981 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Refusing update from remote peer 15073e88db734900ac12653cfa564e88: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251210 14:58:39.848374 14213 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251210 14:58:39.848591 13931 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251210 14:58:39.850380 14213 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e37585beb7a84c379041754b1aceb780 (127.6.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.850888 13981 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e37585beb7a84c379041754b1aceb780 (127.6.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.850950 14171 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e37585beb7a84c379041754b1aceb780 (127.6.115.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.851794 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.851887 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.852002 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.852079 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.852141 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.852187 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.852252 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:39.852277 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:39.852439 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15073e88db734900ac12653cfa564e88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:39.852519 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:40.065711 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.056316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.6.115.125:35771)
1210 14:58:40.056548 (+ 232us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.056551 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.056567 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:58:40.056599 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.056601 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.056639 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.056716 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.057604 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.058131 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.058700 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.058865 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.062732 (+ 3867us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.062748 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.062750 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.062779 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.064275 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.064682 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.064684 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.064686 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.064727 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.064961 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.064963 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.064964 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.065089 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.065181 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.065465 (+ 284us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.065555 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20251210 14:58:40.068661 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.066489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33066)
1210 14:58:40.066712 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.066729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.066817 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.067005 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.067007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.067019 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.067177 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.067181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.067857 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.067859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.068393 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.068400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.068492 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.068505 (+ 13us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.068542 (+ 37us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.068580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20251210 14:58:40.069075 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.066590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.6.115.126:45729)
1210 14:58:40.066833 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.066836 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.066851 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:40.066887 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.066889 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.066926 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.066994 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.067273 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.067767 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.068497 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.068805 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.068842 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.068916 (+ 74us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.068969 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20251210 14:58:40.072252 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.070182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:36502)
1210 14:58:40.070311 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.070327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.070399 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.070609 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.070612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.070626 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.070763 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.070769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.071412 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.071415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.071921 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.071928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.072043 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.072058 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.072109 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.072155 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20251210 14:58:40.072623 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.070263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.6.115.125:35771)
1210 14:58:40.070381 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.070384 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.070396 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:40.070491 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.070494 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.070529 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.070590 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.070871 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.071309 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.072031 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.072442 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.072487 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.072525 (+ 38us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.072561 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:58:40.075717 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.073580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:32896)
1210 14:58:40.073697 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.073711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.073767 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.074109 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.074112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.074125 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.074243 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.074247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.074909 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.074912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.075472 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.075479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.075566 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.075577 (+ 11us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.075611 (+ 34us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.075647 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20251210 14:58:40.076177 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.073690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.6.115.124:41865)
1210 14:58:40.073919 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.073922 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.073935 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:40.073970 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.073972 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.074017 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.074085 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.074327 (+ 242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.074816 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.075574 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.075967 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.075996 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.076045 (+ 49us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.076086 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20251210 14:58:40.083494 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.077697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.6.115.124:41865)
1210 14:58:40.077838 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.077842 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.077855 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:40.077943 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.077946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.077982 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.078046 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.078719 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.079143 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.079757 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.079940 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.080891 (+ 951us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.080905 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.080907 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.080929 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.082388 (+ 1459us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.082750 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.082753 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.082759 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.082792 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.082955 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.082957 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.082958 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.083072 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.083167 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.083318 (+ 151us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.083400 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251210 14:58:40.086661 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.084214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33074)
1210 14:58:40.084444 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.084457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.084552 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.084766 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.084769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.084786 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.084961 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.084967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.085691 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.085694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.086293 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.086301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.086418 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.086434 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.086482 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.086534 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20251210 14:58:40.087073 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.084323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.6.115.126:45729)
1210 14:58:40.084448 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.084454 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.084525 (+ 71us) server_negotiation.cc:408] Connection header received
1210 14:58:40.084642 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.084645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.084683 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.084768 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.085080 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.085587 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.086426 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.086890 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.086917 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.086955 (+ 38us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.086992 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251210 14:58:40.090296 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.088090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.0.0.1:36510)
1210 14:58:40.088334 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.088345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.088402 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.088612 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.088614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.088627 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.088754 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.088759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.089421 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.089423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.090047 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.090055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.090139 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.090150 (+ 11us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.090184 (+ 34us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.090231 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20251210 14:58:40.090718 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.088237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.6.115.125:35771)
1210 14:58:40.088374 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.088377 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.088392 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:40.088486 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.088489 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.088525 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.088604 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.088838 (+ 234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.089289 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.090153 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.090520 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.090558 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.090603 (+ 45us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.090643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251210 14:58:40.093744 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.091702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.0.0.1:32908)
1210 14:58:40.091804 (+ 102us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.091814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.091884 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.092061 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.092063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.092086 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.092232 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.092236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.092858 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.092861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.093404 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.093414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.093510 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.093524 (+ 14us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.093572 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.093617 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":2560}
I20251210 14:58:40.094085 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.091803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.6.115.124:41865)
1210 14:58:40.091891 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.091894 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.091907 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:40.091949 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.091953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.091984 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.092046 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.092319 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.092702 (+ 383us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.093507 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.093930 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.093953 (+ 23us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.093987 (+ 34us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.094021 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":5888}
I20251210 14:58:40.095360 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13423
I20251210 14:58:40.101586 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:38623
--local_ip_for_outbound_sockets=127.6.115.65
--tserver_master_addrs=127.6.115.126:45729
--webserver_port=42167
--webserver_interface=127.6.115.65
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:40.204382 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:40.204569 14224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:40.204619 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:40.207886 14224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:40.207938 14224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:40.208014 14224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:40.211900 14224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:38623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=42167
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:40.212810 14224 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:40.213636 14224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:40.219671 14232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.219640 14229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.219723 14230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:40.219794 14224 server_base.cc:1047] running on GCE node
I20251210 14:58:40.220239 14224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:40.220706 14224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:40.221877 14224 hybrid_clock.cc:648] HybridClock initialized: now 1765378720221855 us; error 38 us; skew 500 ppm
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:40.224553 14224 init.cc:377] Logged in from keytab as oryx/127.6.115.65@KRBTEST.COM (short username oryx)
I20251210 14:58:40.225580 14224 webserver.cc:492] Webserver started at http://127.6.115.65:42167/ using document root <none> and password file <none>
I20251210 14:58:40.226120 14224 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:40.226194 14224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:40.229561 14224 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20251210 14:58:40.231352 14239 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.232368 14224 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:40.232471 14224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "0f17f9a97d6842679ebee2dfca09fd81"
format_stamp: "Formatted at 2025-12-10 14:58:37 on dist-test-slave-nb51"
server_key: "2602b621fddab2db052c2f8ff1fd5d4f"
server_key_iv: "c46f101785e0dc8c14e021c5958b09aa"
server_key_version: "encryptionkey@0"
I20251210 14:58:40.232847 14224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:40.247910 14224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:40.250304 14224 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:40.250466 14224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:40.250943 14224 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:40.251725 14224 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:40.251775 14224 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.251823 14224 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:40.251858 14224 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.260495 14224 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:38623
I20251210 14:58:40.260517 14352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:38623 every 8 connection(s)
I20251210 14:58:40.261379 14224 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:40.266983 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 14224
I20251210 14:58:40.267189 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13559
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:40.275689 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.262976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:50761 (local address 127.6.115.126:45729)
1210 14:58:40.263100 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.263105 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.264450 (+ 1345us) server_negotiation.cc:408] Connection header received
1210 14:58:40.264896 (+ 446us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.264916 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.264968 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.265026 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.266561 (+ 1535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267300 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.268056 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268239 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.271721 (+ 3482us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.271754 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.271757 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.271780 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.273513 (+ 1733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274123 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.274126 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.274128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.274170 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274499 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.274502 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.274504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.274664 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.274745 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.275414 (+ 669us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.275532 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:58:40.275696 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.263473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:34075 (local address 127.6.115.124:41865)
1210 14:58:40.263607 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.263611 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.264243 (+ 632us) server_negotiation.cc:408] Connection header received
1210 14:58:40.264869 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.264873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.264950 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.265029 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.266420 (+ 1391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267135 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.268056 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268239 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.271451 (+ 3212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.271471 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.271477 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.271512 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.273427 (+ 1915us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274123 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.274126 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.274128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.274170 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274499 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.274502 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.274504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.274664 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.274731 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.275417 (+ 686us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.275532 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251210 14:58:40.276458 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:45829
--local_ip_for_outbound_sockets=127.6.115.66
--tserver_master_addrs=127.6.115.126:45729
--webserver_port=33477
--webserver_interface=127.6.115.66
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251210 14:58:40.276952 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.263274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:50761)
1210 14:58:40.264370 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.264379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.264701 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.265209 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.265217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.266169 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.266443 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.266449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267417 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.267420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.267927 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.267934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268434 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.269281 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.269293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.271587 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.273621 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.273624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.273625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.274014 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.274271 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.274276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.274383 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.274814 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.274821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.275087 (+ 266us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.275855 (+ 768us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.276150 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":130,"spinlock_wait_cycles":41856,"thread_start_us":131,"threads_started":1}
I20251210 14:58:40.277101 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.263276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:53075 (local address 127.6.115.125:35771)
1210 14:58:40.263404 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.263409 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.263963 (+ 554us) server_negotiation.cc:408] Connection header received
1210 14:58:40.264895 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.264916 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.264968 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.265021 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.266421 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267046 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.268031 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268247 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.272899 (+ 4652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.272922 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.272928 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.272959 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.274907 (+ 1948us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.275407 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.275413 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.275418 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.275467 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.275753 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.275759 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.275764 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.275951 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.276026 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.276773 (+ 747us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.276895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:58:40.276962 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.263274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.6.115.65:34075)
1210 14:58:40.264148 (+ 874us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.264159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.264694 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.265207 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.265215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.265690 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.266244 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.266262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267308 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.267311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.267903 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.267914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268434 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.269167 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.269186 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.271271 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.273580 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.273587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.273598 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.274014 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.274271 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.274274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.274276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.274381 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.274814 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.274821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.275087 (+ 266us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.275995 (+ 908us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.276220 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":217,"spinlock_wait_cycles":26624,"thread_start_us":89,"threads_started":1}
I20251210 14:58:40.278337 14353 heartbeater.cc:344] Connected to a master server at 127.6.115.124:41865
I20251210 14:58:40.278364 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.263274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.6.115.65:53075)
1210 14:58:40.263840 (+ 566us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.263878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.264690 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.265209 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.265217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.265706 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.266246 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.266262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.267230 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.267236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.267884 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.267897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.268435 (+ 538us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.269563 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.269583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.271657 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.275037 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.275043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.275047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.275285 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.275577 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.275583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.275586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.275637 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.276375 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.276378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.276475 (+ 97us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.277254 (+ 779us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.277408 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":46464,"thread_start_us":121,"threads_started":1}
I20251210 14:58:40.278569 14353 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.278956 14355 heartbeater.cc:344] Connected to a master server at 127.6.115.125:35771
I20251210 14:58:40.279045 14355 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.279129 14353 heartbeater.cc:507] Master 127.6.115.124:41865 requested a full tablet report, sending...
I20251210 14:58:40.279229 14355 heartbeater.cc:507] Master 127.6.115.125:35771 requested a full tablet report, sending...
I20251210 14:58:40.279572 14354 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:40.279639 14354 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.279832 14354 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
I20251210 14:58:40.280532 14161 ts_manager.cc:194] Registered new tserver with Master: 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
I20251210 14:58:40.280534 13970 ts_manager.cc:194] Registered new tserver with Master: 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
I20251210 14:58:40.281383 13359 ts_manager.cc:194] Re-registered known tserver with Master: 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
I20251210 14:58:40.282183 13359 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:50761
W20251210 14:58:40.381732 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:40.381928 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:40.381990 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:40.385241 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:40.385298 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:40.385426 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:40.389535 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:45829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=33477
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:40.390470 14360 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:40.391213 14360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:40.397469 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.397464 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.397464 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:40.397922 14360 server_base.cc:1047] running on GCE node
I20251210 14:58:40.398249 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:40.398785 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:40.399947 14360 hybrid_clock.cc:648] HybridClock initialized: now 1765378720399927 us; error 31 us; skew 500 ppm
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:40.402786 14360 init.cc:377] Logged in from keytab as oryx/127.6.115.66@KRBTEST.COM (short username oryx)
I20251210 14:58:40.403923 14360 webserver.cc:492] Webserver started at http://127.6.115.66:33477/ using document root <none> and password file <none>
I20251210 14:58:40.404502 14360 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:40.404583 14360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:40.408248 14360 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:40.410218 14375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.411228 14360 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251210 14:58:40.411334 14360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "e554247114c24a86a457a3600f421282"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "39bb641a52fba6719c05b282bb9aa62c"
server_key_iv: "b5517e7bce4d569e529c1ac6f80a2efd"
server_key_version: "encryptionkey@0"
I20251210 14:58:40.411741 14360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:40.422230 14360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:40.424736 14360 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:40.424894 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:40.425415 14360 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:40.426214 14360 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:40.426261 14360 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.426316 14360 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:40.426352 14360 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.434900 14360 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:45829
I20251210 14:58:40.434969 14488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:45829 every 8 connection(s)
I20251210 14:58:40.436092 14360 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:40.444496 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 14360
I20251210 14:58:40.444684 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13695
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
I20251210 14:58:40.451485 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.439270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:41109 (local address 127.6.115.124:41865)
1210 14:58:40.439435 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.439441 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.439479 (+ 38us) server_negotiation.cc:408] Connection header received
1210 14:58:40.439852 (+ 373us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.439856 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.439917 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.440018 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.442170 (+ 2152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.442944 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.443772 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443990 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.447106 (+ 3116us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.447126 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.447129 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.447157 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.449630 (+ 2473us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450108 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.450111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.450114 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.450163 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450459 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.450464 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.450469 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.450649 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.450730 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.451201 (+ 471us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.451323 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251210 14:58:40.451572 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.440436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:36027 (local address 127.6.115.126:45729)
1210 14:58:40.440588 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.440593 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.440609 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:58:40.440670 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.440673 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.440728 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.441137 (+ 409us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.442043 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.442737 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.443360 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443536 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.447108 (+ 3572us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.447131 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.447138 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.447171 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.449201 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.449845 (+ 644us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.449852 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.449856 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.449918 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450257 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.450263 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.450268 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.450472 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.450557 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.451209 (+ 652us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.451347 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251210 14:58:40.451660 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.440574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:39237 (local address 127.6.115.125:35771)
1210 14:58:40.440690 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.440694 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.440999 (+ 305us) server_negotiation.cc:408] Connection header received
1210 14:58:40.441171 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.441175 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.441231 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.441322 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.442526 (+ 1204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443291 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.444109 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.444312 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.448009 (+ 3697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.448034 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.448039 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.448072 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.450126 (+ 2054us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450545 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.450549 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.450550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.450591 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450860 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.450863 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.450865 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.451039 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.451113 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.451418 (+ 305us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.451525 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:58:40.452843 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.438287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:36027)
1210 14:58:40.440224 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.440239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.440322 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.441275 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.441281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.441526 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.441870 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.441884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.442851 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.442853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.443242 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.443247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443435 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.444160 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.444204 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.446926 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.449342 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.449350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.449391 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.449727 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.450034 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.450045 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.450158 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.450625 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.450634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.450912 (+ 278us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.452250 (+ 1338us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.452401 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"mutex_wait_us":25,"spinlock_wait_cycles":20736,"thread_start_us":144,"threads_started":1}
I20251210 14:58:40.452818 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.438287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.6.115.66:41109)
1210 14:58:40.438796 (+ 509us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.438836 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.439640 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.440175 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.440187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.440767 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.442023 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.442033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443086 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.443089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.443657 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.443667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443782 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.444442 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.444454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.446941 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.449777 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.449781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.449783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.449990 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.450270 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.450280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.450343 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.450797 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.450803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.450919 (+ 116us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.451707 (+ 788us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.451999 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"spinlock_wait_cycles":896,"thread_start_us":124,"threads_started":1}
I20251210 14:58:40.452837 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.439934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.6.115.66:39237)
1210 14:58:40.440916 (+ 982us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.440931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.441059 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.441866 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.441869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.442102 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.442394 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.442401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.443435 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.443439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.443995 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.444007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.444105 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.446409 (+ 2304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.446425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.447843 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.450238 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.450244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.450447 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.450700 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.450706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.450708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.450761 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.451178 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.451181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.451249 (+ 68us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.451814 (+ 565us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.452071 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":114,"spinlock_wait_cycles":6016,"thread_start_us":83,"threads_started":1}
I20251210 14:58:40.453699 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:43469
--local_ip_for_outbound_sockets=127.6.115.67
--tserver_master_addrs=127.6.115.126:45729
--webserver_port=33935
--webserver_interface=127.6.115.67
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865 with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251210 14:58:40.454407 14489 heartbeater.cc:344] Connected to a master server at 127.6.115.124:41865
I20251210 14:58:40.454605 14489 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.455057 14489 heartbeater.cc:507] Master 127.6.115.124:41865 requested a full tablet report, sending...
I20251210 14:58:40.455682 14491 heartbeater.cc:344] Connected to a master server at 127.6.115.125:35771
I20251210 14:58:40.455765 14491 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.455968 14490 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:40.455971 14491 heartbeater.cc:507] Master 127.6.115.125:35771 requested a full tablet report, sending...
I20251210 14:58:40.456074 14490 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.456274 14161 ts_manager.cc:194] Registered new tserver with Master: e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:40.456288 14490 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
I20251210 14:58:40.456811 13970 ts_manager.cc:194] Registered new tserver with Master: e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:40.457194 13359 ts_manager.cc:194] Re-registered known tserver with Master: e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:40.457805 13359 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:36027
W20251210 14:58:40.557276 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:40.557546 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:40.557634 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:40.560803 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:40.560865 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:40.560982 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:40.565186 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:43469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=33935
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:45729,127.6.115.125:35771,127.6.115.124:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:40.566190 14496 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:40.567009 14496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:40.573312 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.573343 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:40.573333 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:40.573570 14496 server_base.cc:1047] running on GCE node
I20251210 14:58:40.573971 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:40.574451 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:40.575603 14496 hybrid_clock.cc:648] HybridClock initialized: now 1765378720575562 us; error 55 us; skew 500 ppm
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:40.578172 14496 init.cc:377] Logged in from keytab as oryx/127.6.115.67@KRBTEST.COM (short username oryx)
I20251210 14:58:40.579080 14496 webserver.cc:492] Webserver started at http://127.6.115.67:33935/ using document root <none> and password file <none>
I20251210 14:58:40.579558 14496 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:40.579610 14496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:40.582888 14496 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:40.584651 14511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.585780 14496 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:40.585880 14496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "27193df16bea4b3689ad1f635c99d1f4"
format_stamp: "Formatted at 2025-12-10 14:58:38 on dist-test-slave-nb51"
server_key: "a0025272fcd596152febb9264ce443c8"
server_key_iv: "f51671bcfc9ece490962f6e31e0e6190"
server_key_version: "encryptionkey@0"
I20251210 14:58:40.586262 14496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:40.604703 14496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:40.607553 14496 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:40.607739 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:40.608245 14496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:40.609098 14496 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:40.609144 14496 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.609198 14496 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:40.609230 14496 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:40.618312 14496 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:43469
I20251210 14:58:40.618340 14624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:43469 every 8 connection(s)
I20251210 14:58:40.619218 14496 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:40.620604 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 14496
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.125@KRBTEST.COM
Dec 10 14:58:40 dist-test-slave-nb51 krb5kdc[13311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378720, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.124@KRBTEST.COM
I20251210 14:58:40.634752 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.622894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.6.115.125:35771)
1210 14:58:40.623992 (+ 1098us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.623997 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.624026 (+ 29us) server_negotiation.cc:408] Connection header received
1210 14:58:40.624061 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.624064 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.624104 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.624197 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.625271 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.626009 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.628961 (+ 2952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.629143 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.629225 (+ 82us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.629241 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.629244 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.629261 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.631749 (+ 2488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.632222 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.632259 (+ 37us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.632261 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.632313 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.632589 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.632595 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.632598 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.632814 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.632951 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.633214 (+ 263us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.634510 (+ 1296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":69,"threads_started":1}
I20251210 14:58:40.636679 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.622338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:42245 (local address 127.6.115.125:35771)
1210 14:58:40.622495 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.622499 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.623447 (+ 948us) server_negotiation.cc:408] Connection header received
1210 14:58:40.623515 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.623518 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.623586 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.623692 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.625151 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.625969 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.626966 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.627179 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.631277 (+ 4098us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.631293 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.631297 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.631324 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.634207 (+ 2883us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.634845 (+ 638us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.634849 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.634851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.634905 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.635408 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.635411 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.635413 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.635581 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.635664 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.636354 (+ 690us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.636485 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251210 14:58:40.637609 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.623101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.6.115.126:45729)
1210 14:58:40.626502 (+ 3401us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.626508 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.626523 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:40.626563 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.626566 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.626616 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.626711 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.627797 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.628660 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.629792 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.630020 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.631148 (+ 1128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.631168 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.631172 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.631199 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.635464 (+ 4265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.635944 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.635949 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.635951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.635997 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.636255 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.636258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.636259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.636444 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.636559 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.637269 (+ 710us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.637415 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3322,"thread_start_us":70,"threads_started":1}
I20251210 14:58:40.639173 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.623335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.6.115.124:41865)
1210 14:58:40.623707 (+ 372us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.623712 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.624604 (+ 892us) server_negotiation.cc:408] Connection header received
1210 14:58:40.624808 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.624814 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.624874 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.624951 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.628878 (+ 3927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.629695 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.631477 (+ 1782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.631710 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.634138 (+ 2428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.634161 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.634166 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.634196 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.637295 (+ 3099us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.637851 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.637856 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.637858 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.637921 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638213 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.638216 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.638218 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.638381 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.638496 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.638839 (+ 343us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.638979 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":77,"threads_started":1}
I20251210 14:58:40.639639 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.622172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.125:35771 (local address 127.6.115.67:42245)
1210 14:58:40.623343 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.623355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.623440 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.623828 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.623834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.624698 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.624984 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.625003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.626165 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.626169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.626824 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.626833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.627317 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.629486 (+ 2169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.629504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.631105 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.634355 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.634361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.634373 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.634707 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.635175 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.635178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.635180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.635292 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.635756 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.635763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.636043 (+ 280us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.637090 (+ 1047us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.637426 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":139,"threads_started":1}
I20251210 14:58:40.639943 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.621010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:57611 (local address 127.6.115.126:45729)
1210 14:58:40.621166 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.621171 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.621782 (+ 611us) server_negotiation.cc:408] Connection header received
1210 14:58:40.622917 (+ 1135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.622924 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.622984 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.623040 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.625006 (+ 1966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.625796 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.626684 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.626887 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.633889 (+ 7002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.633905 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.633911 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.633941 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.636278 (+ 2337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638613 (+ 2335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.638618 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.638623 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.638671 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638983 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.638986 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.638988 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.639145 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.639219 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.639719 (+ 500us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.639824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251210 14:58:40.640499 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.621206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:57611)
1210 14:58:40.621627 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.621673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.622685 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.623430 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.623439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.623970 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.624830 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.624848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.625936 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.625939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.626537 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.626546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.627111 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.627882 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.627921 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.632380 (+ 4459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.638200 (+ 5820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.638210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.638495 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.638794 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.638799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.638880 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.639413 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.639419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.639513 (+ 94us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.640228 (+ 715us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.640353 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"spinlock_wait_cycles":9472,"thread_start_us":116,"threads_started":1}
I20251210 14:58:40.641023 14627 heartbeater.cc:344] Connected to a master server at 127.6.115.125:35771
I20251210 14:58:40.641275 14627 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.641913 14627 heartbeater.cc:507] Master 127.6.115.125:35771 requested a full tablet report, sending...
I20251210 14:58:40.642098 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.623151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:60115 (local address 127.6.115.124:41865)
1210 14:58:40.623293 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.623299 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.626317 (+ 3018us) server_negotiation.cc:408] Connection header received
1210 14:58:40.627974 (+ 1657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.627980 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.628037 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.628101 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.630616 (+ 2515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.631310 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.632186 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.632412 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.636091 (+ 3679us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.636108 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.636112 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.636137 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.638181 (+ 2044us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638657 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.638660 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.638663 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.638705 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638965 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.638970 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.638971 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.641183 (+ 2212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.641278 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.641865 (+ 587us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.641971 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251210 14:58:40.642287 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.622800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.124:41865 (local address 127.6.115.67:60115)
1210 14:58:40.626238 (+ 3438us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.626252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.627823 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.628250 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.628253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.628426 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:40.628706 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.628719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.631453 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.631458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.632033 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.632046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.632523 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.633601 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:40.633640 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:40.635945 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:40.638287 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.638293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.638555 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.638802 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:40.638805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:40.638806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:40.638851 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:40.641562 (+ 2711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:40.641566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:40.641652 (+ 86us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.641768 (+ 116us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.641893 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3363,"thread_start_us":87,"threads_started":1}
I20251210 14:58:40.640441 13359 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251210 14:58:40.642982 14626 heartbeater.cc:344] Connected to a master server at 127.6.115.126:45729
I20251210 14:58:40.643013 14625 heartbeater.cc:344] Connected to a master server at 127.6.115.124:41865
I20251210 14:58:40.643072 14626 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.643072 14625 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:40.643121 13970 ts_manager.cc:194] Registered new tserver with Master: 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67:43469)
I20251210 14:58:40.643316 14625 heartbeater.cc:507] Master 127.6.115.124:41865 requested a full tablet report, sending...
I20251210 14:58:40.643313 14626 heartbeater.cc:507] Master 127.6.115.126:45729 requested a full tablet report, sending...
W20251210 14:58:40.643478 13359 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:40.644039 13358 ts_manager.cc:194] Re-registered known tserver with Master: 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67:43469)
I20251210 14:58:40.644057 14161 ts_manager.cc:194] Registered new tserver with Master: 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67:43469)
I20251210 14:58:40.644697 13358 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:57611
I20251210 14:58:40.659271 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.655057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43469 (local address 127.0.0.1:60160)
1210 14:58:40.655326 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.655341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.655417 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.656309 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.656312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.656325 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.656538 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.656543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.657970 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.657973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.658885 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.658894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.659005 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.659023 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.659068 (+ 45us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.659128 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
I20251210 14:58:40.659547 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.655448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:38623 (local address 127.0.0.1:42666)
1210 14:58:40.656074 (+ 626us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.656090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.656166 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.656623 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.656629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.656651 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.656919 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.656926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.658289 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.658292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.659093 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.659100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.659184 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.659197 (+ 13us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.659240 (+ 43us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.659286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":103,"threads_started":1}
I20251210 14:58:40.659761 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.654721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:45829 (local address 127.0.0.1:40582)
1210 14:58:40.654989 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.655007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.655091 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.657077 (+ 1986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.657080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.657099 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.657319 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.657328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.658540 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.658543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.659225 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.659235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.659342 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.659360 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.659410 (+ 50us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.659465 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20251210 14:58:40.660056 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.655434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.6.115.67:43469)
1210 14:58:40.655872 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.655879 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.655904 (+ 25us) server_negotiation.cc:408] Connection header received
1210 14:58:40.655980 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.655986 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.656173 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.656325 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.656686 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.657838 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.659025 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.659660 (+ 635us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.659781 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.659843 (+ 62us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.659928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":115,"threads_started":1}
I20251210 14:58:40.660521 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.654874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.6.115.66:45829)
1210 14:58:40.656639 (+ 1765us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.656645 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.656666 (+ 21us) server_negotiation.cc:408] Connection header received
1210 14:58:40.656759 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.656764 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.656889 (+ 125us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.657077 (+ 188us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.657464 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.658420 (+ 956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.659622 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.660141 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.660251 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.660310 (+ 59us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.660394 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1639,"thread_start_us":111,"threads_started":1}
I20251210 14:58:40.660810 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.655865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.6.115.65:38623)
1210 14:58:40.656158 (+ 293us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.656165 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.656187 (+ 22us) server_negotiation.cc:408] Connection header received
1210 14:58:40.656274 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.656282 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.656464 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.656625 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.657045 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.658178 (+ 1133us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.660034 (+ 1856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.660475 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.660546 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.660605 (+ 59us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.660698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20251210 14:58:40.662837 14559 tablet_service.cc:1505] Processing CreateTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 (DEFAULT_TABLE table=test-table [id=1208111425b84c6fb492aeab3a18a150]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:40.663100 14423 tablet_service.cc:1505] Processing CreateTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 (DEFAULT_TABLE table=test-table [id=1208111425b84c6fb492aeab3a18a150]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:40.663098 14287 tablet_service.cc:1505] Processing CreateTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 (DEFAULT_TABLE table=test-table [id=1208111425b84c6fb492aeab3a18a150]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:40.663614 14559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90654cb0e6fb4c70b44a5751cd0a76f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:40.663924 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90654cb0e6fb4c70b44a5751cd0a76f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:40.663918 14287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90654cb0e6fb4c70b44a5751cd0a76f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:40.669270 14656 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Bootstrap starting.
I20251210 14:58:40.670662 14657 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Bootstrap starting.
I20251210 14:58:40.672366 14656 tablet_bootstrap.cc:654] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:40.672664 14657 tablet_bootstrap.cc:654] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:40.673235 14656 log.cc:826] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:40.673686 14658 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Bootstrap starting.
I20251210 14:58:40.675263 14656 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: No bootstrap required, opened a new log
I20251210 14:58:40.675447 14656 ts_tablet_manager.cc:1403] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251210 14:58:40.675581 14658 tablet_bootstrap.cc:654] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:40.676319 14658 log.cc:826] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:40.676712 14657 log.cc:826] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:40.678011 14658 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: No bootstrap required, opened a new log
I20251210 14:58:40.678179 14658 ts_tablet_manager.cc:1403] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:58:40.678443 14657 tablet_bootstrap.cc:492] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: No bootstrap required, opened a new log
I20251210 14:58:40.678473 14656 raft_consensus.cc:359] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.678645 14657 ts_tablet_manager.cc:1403] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251210 14:58:40.678697 14656 raft_consensus.cc:385] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:40.678755 14656 raft_consensus.cc:740] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27193df16bea4b3689ad1f635c99d1f4, State: Initialized, Role: FOLLOWER
I20251210 14:58:40.679189 14656 consensus_queue.cc:260] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.680015 14656 ts_tablet_manager.cc:1434] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251210 14:58:40.680191 14626 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:40.681433 14658 raft_consensus.cc:359] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.681624 14658 raft_consensus.cc:385] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:40.681531 14657 raft_consensus.cc:359] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.681676 14658 raft_consensus.cc:740] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f17f9a97d6842679ebee2dfca09fd81, State: Initialized, Role: FOLLOWER
I20251210 14:58:40.681689 14657 raft_consensus.cc:385] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:40.681735 14657 raft_consensus.cc:740] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e554247114c24a86a457a3600f421282, State: Initialized, Role: FOLLOWER
I20251210 14:58:40.682374 14658 consensus_queue.cc:260] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.683199 14658 ts_tablet_manager.cc:1434] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251210 14:58:40.683432 14657 consensus_queue.cc:260] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.684159 14657 ts_tablet_manager.cc:1434] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:58:40.684409 14354 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:40.684456 14490 heartbeater.cc:499] Master 127.6.115.126:45729 was elected leader, sending a full tablet report...
I20251210 14:58:40.685981 14662 raft_consensus.cc:493] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:40.686129 14662 raft_consensus.cc:515] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.687112 14662 leader_election.cc:290] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e554247114c24a86a457a3600f421282 (127.6.115.66:45829), 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
W20251210 14:58:40.688190 14492 tablet.cc:2378] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:40.691218 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.687251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:45829 (local address 127.6.115.67:58899)
1210 14:58:40.687618 (+ 367us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.687633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.687723 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.688654 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.688657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.688674 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.688943 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.688950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.689946 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.689949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.690866 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.690876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.690977 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.690994 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.691039 (+ 45us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.691090 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"spinlock_wait_cycles":6912}
I20251210 14:58:40.691906 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.688357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:33571 (local address 127.6.115.65:38623)
1210 14:58:40.688464 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.688468 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.688480 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:40.688518 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.688521 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.688560 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.688651 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.689078 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.689784 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.691181 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.691658 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.691695 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.691742 (+ 47us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.691795 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:58:40.692183 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.687346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:58899 (local address 127.6.115.66:45829)
1210 14:58:40.687464 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.687470 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.688040 (+ 570us) server_negotiation.cc:408] Connection header received
1210 14:58:40.688107 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.688110 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.688153 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.688221 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:40.689068 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.689829 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.691471 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.691959 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.691996 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.692039 (+ 43us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.692086 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251210 14:58:40.692659 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90654cb0e6fb4c70b44a5751cd0a76f1" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554247114c24a86a457a3600f421282" is_pre_election: true
I20251210 14:58:40.692883 14443 raft_consensus.cc:2468] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 0.
I20251210 14:58:40.693540 14513 leader_election.cc:304] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27193df16bea4b3689ad1f635c99d1f4, e554247114c24a86a457a3600f421282; no voters:
I20251210 14:58:40.693907 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.687482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:38623 (local address 127.6.115.67:33571)
1210 14:58:40.687928 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:40.687940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:40.688009 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:40.688665 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:40.688668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:40.688689 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:40.688946 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.688955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.689905 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.689908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:40.690814 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:40.690823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.690938 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.690959 (+ 21us) client_negotiation.cc:769] Sending connection context
1210 14:58:40.691017 (+ 58us) client_negotiation.cc:240] Negotiation successful
1210 14:58:40.691090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"spinlock_wait_cycles":1408}
I20251210 14:58:40.694298 14662 raft_consensus.cc:2804] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:40.694373 14662 raft_consensus.cc:493] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:40.694415 14662 raft_consensus.cc:3060] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:40.694494 14307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90654cb0e6fb4c70b44a5751cd0a76f1" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f17f9a97d6842679ebee2dfca09fd81" is_pre_election: true
I20251210 14:58:40.694690 14307 raft_consensus.cc:2468] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 0.
I20251210 14:58:40.696008 14662 raft_consensus.cc:515] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.696360 14662 leader_election.cc:290] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 election: Requested vote from peers e554247114c24a86a457a3600f421282 (127.6.115.66:45829), 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623)
I20251210 14:58:40.696768 14307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90654cb0e6fb4c70b44a5751cd0a76f1" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f17f9a97d6842679ebee2dfca09fd81"
I20251210 14:58:40.696800 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90654cb0e6fb4c70b44a5751cd0a76f1" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554247114c24a86a457a3600f421282"
I20251210 14:58:40.696866 14307 raft_consensus.cc:3060] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:40.696904 14443 raft_consensus.cc:3060] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:40.697940 14307 raft_consensus.cc:2468] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 1.
I20251210 14:58:40.698081 14443 raft_consensus.cc:2468] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 1.
I20251210 14:58:40.698338 14513 leader_election.cc:304] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27193df16bea4b3689ad1f635c99d1f4, e554247114c24a86a457a3600f421282; no voters:
I20251210 14:58:40.698491 14662 raft_consensus.cc:2804] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:40.698712 14662 raft_consensus.cc:697] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 LEADER]: Becoming Leader. State: Replica: 27193df16bea4b3689ad1f635c99d1f4, State: Running, Role: LEADER
I20251210 14:58:40.699014 14662 consensus_queue.cc:237] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } }
I20251210 14:58:40.702790 13359 catalog_manager.cc:5654] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "27193df16bea4b3689ad1f635c99d1f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:40.726064 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.723229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.6.115.67:43469)
1210 14:58:40.723354 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.723358 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.723370 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:40.723441 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.723443 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.723496 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.723576 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:40.723855 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.724333 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.724907 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.725045 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.725130 (+ 85us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:40.725652 (+ 522us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:40.725769 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.725869 (+ 100us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.725918 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251210 14:58:40.731114 14443 raft_consensus.cc:1275] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Refusing update from remote peer 27193df16bea4b3689ad1f635c99d1f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:40.731168 14307 raft_consensus.cc:1275] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Refusing update from remote peer 27193df16bea4b3689ad1f635c99d1f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:40.731768 14665 consensus_queue.cc:1048] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:40.731930 14662 consensus_queue.cc:1048] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:40.739661 14670 mvcc.cc:204] Tried to move back new op lower bound from 7230991240108208128 to 7230991239986401280. Current Snapshot: MvccSnapshot[applied={T|T < 7230991240108208128}]
I20251210 14:58:40.743044 14669 mvcc.cc:204] Tried to move back new op lower bound from 7230991240108208128 to 7230991239986401280. Current Snapshot: MvccSnapshot[applied={T|T < 7230991240108208128 or (T in {7230991240108208128})}]
I20251210 14:58:40.749408 13359 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:40.749565 13359 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:40.754220 13359 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 15073e88db734900ac12653cfa564e88: Sending DeleteTablet for 3 replicas of tablet 90654cb0e6fb4c70b44a5751cd0a76f1
I20251210 14:58:40.754956 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:40 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:40582
I20251210 14:58:40.755100 14559 tablet_service.cc:1552] Processing DeleteTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:40 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:60160
I20251210 14:58:40.755118 14287 tablet_service.cc:1552] Processing DeleteTablet for tablet 90654cb0e6fb4c70b44a5751cd0a76f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:40 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:42666
I20251210 14:58:40.755563 14679 tablet_replica.cc:333] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: stopping tablet replica
I20251210 14:58:40.756382 14679 raft_consensus.cc:2243] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:40.756603 14678 tablet_replica.cc:333] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: stopping tablet replica
I20251210 14:58:40.756646 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13327
I20251210 14:58:40.757511 14678 raft_consensus.cc:2243] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:40.757813 14679 raft_consensus.cc:2272] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:40.757863 14678 raft_consensus.cc:2272] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:40.759295 14678 ts_tablet_manager.cc:1916] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:40.759306 14679 ts_tablet_manager.cc:1916] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:40.761574 14680 tablet_replica.cc:333] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: stopping tablet replica
I20251210 14:58:40.761865 14680 raft_consensus.cc:2243] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:40.762184 14680 raft_consensus.cc:2272] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:40.762300 14678 ts_tablet_manager.cc:1929] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:40.762444 14678 log.cc:1199] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/90654cb0e6fb4c70b44a5751cd0a76f1
I20251210 14:58:40.762915 14678 ts_tablet_manager.cc:1950] T 90654cb0e6fb4c70b44a5751cd0a76f1 P e554247114c24a86a457a3600f421282: Deleting consensus metadata
I20251210 14:58:40.763751 14679 ts_tablet_manager.cc:1929] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:40.763849 14679 log.cc:1199] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/90654cb0e6fb4c70b44a5751cd0a76f1
I20251210 14:58:40.764181 14679 ts_tablet_manager.cc:1950] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 27193df16bea4b3689ad1f635c99d1f4: Deleting consensus metadata
W20251210 14:58:40.765226 14513 connection.cc:441] server connection from 127.0.0.1:60160 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60160 (request call id 1) could send its response
W20251210 14:58:40.765527 14379 connection.cc:537] server connection from 127.0.0.1:40582 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251210 14:58:40.766171 14680 ts_tablet_manager.cc:1916] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:40.768796 14680 ts_tablet_manager.cc:1929] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:40.768908 14680 log.cc:1199] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/90654cb0e6fb4c70b44a5751cd0a76f1
I20251210 14:58:40.769213 14680 ts_tablet_manager.cc:1950] T 90654cb0e6fb4c70b44a5751cd0a76f1 P 0f17f9a97d6842679ebee2dfca09fd81: Deleting consensus metadata
W20251210 14:58:40.770148 14242 connection.cc:441] server connection from 127.0.0.1:42666 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42666 (request call id 1) could send its response
I20251210 14:58:40.777695 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.768484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.6.115.125:35771)
1210 14:58:40.768619 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.768627 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.768642 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:40.768678 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.768680 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.768720 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.768797 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.770188 (+ 1391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.770925 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.771816 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.772008 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.773923 (+ 1915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.773943 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.773947 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.773976 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.775973 (+ 1997us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.776440 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.776444 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.776446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.776498 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.776755 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.776759 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.776762 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.776923 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.777068 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.777309 (+ 241us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.777511 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251210 14:58:40.779407 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:40.769576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.6.115.124:41865)
1210 14:58:40.769714 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:40.769719 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:40.770481 (+ 762us) server_negotiation.cc:408] Connection header received
1210 14:58:40.770571 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:40.770574 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:40.770622 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:40.770724 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:40.771673 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.772298 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:40.773433 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:40.773963 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:40.775307 (+ 1344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:40.775324 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:40.775327 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:40.775352 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:40.777502 (+ 2150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.778247 (+ 745us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.778251 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.778253 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.778299 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:40.778554 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:40.778557 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:40.778559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:40.778710 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:40.778823 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:40.779131 (+ 308us) server_negotiation.cc:300] Negotiation successful
1210 14:58:40.779225 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251210 14:58:41.743046 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:41.742585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:39129)
1210 14:58:41.742846 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:41.742932 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20251210 14:58:41.743279 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:41.742877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:37673)
1210 14:58:41.743100 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:41.743175 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":57,"threads_started":1}
W20251210 14:58:41.743510 14626 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:45729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
W20251210 14:58:41.743719 14354 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:45729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:41.744064 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:41.743667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:44679)
1210 14:58:41.743915 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:41.743985 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":59,"threads_started":1}
W20251210 14:58:41.744273 14490 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:45729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.254603 14691 raft_consensus.cc:493] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15073e88db734900ac12653cfa564e88)
I20251210 14:58:42.254796 14691 raft_consensus.cc:515] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:42.255052 14692 raft_consensus.cc:493] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15073e88db734900ac12653cfa564e88)
I20251210 14:58:42.255201 14692 raft_consensus.cc:515] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:42.256016 14691 leader_election.cc:290] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), e37585beb7a84c379041754b1aceb780 (127.6.115.124:41865)
I20251210 14:58:42.256165 14692 leader_election.cc:290] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125:35771)
I20251210 14:58:42.256549 13981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e37585beb7a84c379041754b1aceb780" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b70b290d8d4e444fae0462d65682bfa4" is_pre_election: true
I20251210 14:58:42.256560 14171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b70b290d8d4e444fae0462d65682bfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e37585beb7a84c379041754b1aceb780" is_pre_election: true
I20251210 14:58:42.256726 14171 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70b290d8d4e444fae0462d65682bfa4 in term 1.
I20251210 14:58:42.256735 13981 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e37585beb7a84c379041754b1aceb780 in term 1.
I20251210 14:58:42.256760 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.256194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33566)
1210 14:58:42.256480 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.256549 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":60,"threads_started":1}
I20251210 14:58:42.256850 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.256463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33572)
1210 14:58:42.256695 (+ 232us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.256762 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
W20251210 14:58:42.256893 13957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.257073 13958 leader_election.cc:304] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70b290d8d4e444fae0462d65682bfa4, e37585beb7a84c379041754b1aceb780; no voters:
I20251210 14:58:42.257141 14145 leader_election.cc:304] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70b290d8d4e444fae0462d65682bfa4, e37585beb7a84c379041754b1aceb780; no voters:
W20251210 14:58:42.257262 14147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.257328 14691 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251210 14:58:42.257404 14691 raft_consensus.cc:493] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 15073e88db734900ac12653cfa564e88)
I20251210 14:58:42.257421 14692 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251210 14:58:42.257485 14692 raft_consensus.cc:493] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 15073e88db734900ac12653cfa564e88)
I20251210 14:58:42.257573 14692 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 1 FOLLOWER]: Advancing to term 2
I20251210 14:58:42.257589 14691 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 1 FOLLOWER]: Advancing to term 2
I20251210 14:58:42.257997 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.257749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33588)
1210 14:58:42.257887 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.257936 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20251210 14:58:42.258150 13957 leader_election.cc:336] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.258250 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.257986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33594)
1210 14:58:42.258133 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.258183 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20251210 14:58:42.258325 14147 leader_election.cc:336] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.258827 14692 raft_consensus.cc:515] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:42.258922 14691 raft_consensus.cc:515] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:42.259156 14692 leader_election.cc:290] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 election: Requested vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), b70b290d8d4e444fae0462d65682bfa4 (127.6.115.125:35771)
I20251210 14:58:42.259459 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.259260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33600)
1210 14:58:42.259355 (+ 95us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.259400 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20251210 14:58:42.259474 13981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e37585beb7a84c379041754b1aceb780" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b70b290d8d4e444fae0462d65682bfa4"
I20251210 14:58:42.259617 13981 raft_consensus.cc:2393] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e37585beb7a84c379041754b1aceb780 in current term 2: Already voted for candidate b70b290d8d4e444fae0462d65682bfa4 in this term.
I20251210 14:58:42.259689 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.259392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33604)
1210 14:58:42.259576 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.259627 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20251210 14:58:42.259984 14691 leader_election.cc:290] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 election: Requested vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), e37585beb7a84c379041754b1aceb780 (127.6.115.124:41865)
I20251210 14:58:42.260262 14171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b70b290d8d4e444fae0462d65682bfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e37585beb7a84c379041754b1aceb780"
I20251210 14:58:42.260358 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.260105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33614)
1210 14:58:42.260244 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.260292 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251210 14:58:42.260416 14171 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b70b290d8d4e444fae0462d65682bfa4 in current term 2: Already voted for candidate e37585beb7a84c379041754b1aceb780 in this term.
I20251210 14:58:42.260414 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.260227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33616)
1210 14:58:42.260317 (+ 90us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.260363 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251210 14:58:42.260484 13957 leader_election.cc:336] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.260684 13958 leader_election.cc:304] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b70b290d8d4e444fae0462d65682bfa4; no voters: 15073e88db734900ac12653cfa564e88, e37585beb7a84c379041754b1aceb780
I20251210 14:58:42.260897 14691 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251210 14:58:42.261085 14147 leader_election.cc:336] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:42.261150 14147 leader_election.cc:304] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e37585beb7a84c379041754b1aceb780; no voters: 15073e88db734900ac12653cfa564e88, b70b290d8d4e444fae0462d65682bfa4
I20251210 14:58:42.261272 14692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251210 14:58:42.744647 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.744070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:44637)
1210 14:58:42.744437 (+ 367us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.744505 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":92,"threads_started":1}
I20251210 14:58:42.744724 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.744239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:42333)
1210 14:58:42.744524 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.744585 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20251210 14:58:42.745126 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:42.744711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:40185)
1210 14:58:42.744986 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:42.745040 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20251210 14:58:43.745997 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:43.745350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:49657)
1210 14:58:43.745821 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:43.745884 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":78,"threads_started":1}
I20251210 14:58:43.746032 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:43.745489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:52439)
1210 14:58:43.745901 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:43.745947 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":69,"threads_started":1}
I20251210 14:58:43.746215 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:43.745704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:51693)
1210 14:58:43.746023 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:43.746091 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20251210 14:58:44.382694 14703 raft_consensus.cc:493] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:44.382850 14703 raft_consensus.cc:515] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:44.383399 14703 leader_election.cc:290] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), e37585beb7a84c379041754b1aceb780 (127.6.115.124:41865)
I20251210 14:58:44.383925 14171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b70b290d8d4e444fae0462d65682bfa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e37585beb7a84c379041754b1aceb780" is_pre_election: true
I20251210 14:58:44.384027 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.383583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33936)
1210 14:58:44.383847 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.383913 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20251210 14:58:44.384093 14171 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70b290d8d4e444fae0462d65682bfa4 in term 2.
I20251210 14:58:44.384450 13958 leader_election.cc:304] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70b290d8d4e444fae0462d65682bfa4, e37585beb7a84c379041754b1aceb780; no voters:
I20251210 14:58:44.384618 14703 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251210 14:58:44.384699 14703 raft_consensus.cc:493] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:44.384737 14703 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 2 FOLLOWER]: Advancing to term 3
I20251210 14:58:44.384955 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.384683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33950)
1210 14:58:44.384823 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.384876 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20251210 14:58:44.385113 13957 leader_election.cc:336] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:44.385879 14703 raft_consensus.cc:515] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:44.386193 14703 leader_election.cc:290] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 election: Requested vote from peers 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729), e37585beb7a84c379041754b1aceb780 (127.6.115.124:41865)
I20251210 14:58:44.386514 14171 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b70b290d8d4e444fae0462d65682bfa4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e37585beb7a84c379041754b1aceb780"
I20251210 14:58:44.386555 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.386356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33954)
1210 14:58:44.386473 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.386513 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20251210 14:58:44.386647 14171 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 2 FOLLOWER]: Advancing to term 3
I20251210 14:58:44.387163 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.387004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:33968)
1210 14:58:44.387093 (+ 89us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.387126 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251210 14:58:44.387256 13957 leader_election.cc:336] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
I20251210 14:58:44.387740 14171 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70b290d8d4e444fae0462d65682bfa4 in term 3.
I20251210 14:58:44.388087 13958 leader_election.cc:304] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b70b290d8d4e444fae0462d65682bfa4, e37585beb7a84c379041754b1aceb780; no voters: 15073e88db734900ac12653cfa564e88
I20251210 14:58:44.388322 14703 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 3 FOLLOWER]: Leader election won for term 3
I20251210 14:58:44.388509 14703 raft_consensus.cc:697] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [term 3 LEADER]: Becoming Leader. State: Replica: b70b290d8d4e444fae0462d65682bfa4, State: Running, Role: LEADER
I20251210 14:58:44.388747 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } }
I20251210 14:58:44.390057 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b70b290d8d4e444fae0462d65682bfa4. Latest consensus state: current_term: 3 leader_uuid: "b70b290d8d4e444fae0462d65682bfa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:44.390199 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:44.390514 14708 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:44.392467 14708 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:44.392786 14708 catalog_manager.cc:1269] Loaded cluster ID: 2a088d2cbc5f4ff9b8e0f24e0e789f20
I20251210 14:58:44.392827 14708 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:44.393293 14708 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:44.393699 14708 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Loaded TSK: 0
I20251210 14:58:44.394354 14708 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:58:44.478928 13971 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:44.481242 13971 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:44.484053 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.483777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:34000)
1210 14:58:44.483917 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.483974 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20251210 14:58:44.484074 14171 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [term 3 FOLLOWER]: Refusing update from remote peer b70b290d8d4e444fae0462d65682bfa4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251210 14:58:44.484557 14706 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251210 14:58:44.484790 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.484576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.0.0.1:34004)
1210 14:58:44.484684 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.484734 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251210 14:58:44.484920 13957 consensus_peers.cc:597] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 -> Peer 15073e88db734900ac12653cfa564e88 (127.6.115.126:45729): Couldn't send request to peer 15073e88db734900ac12653cfa564e88. Status: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251210 14:58:44.486107 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b70b290d8d4e444fae0462d65682bfa4. Latest consensus state: current_term: 3 leader_uuid: "b70b290d8d4e444fae0462d65682bfa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:44.486249 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:44.487267 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b70b290d8d4e444fae0462d65682bfa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:44.487365 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:44.487617 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b70b290d8d4e444fae0462d65682bfa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15073e88db734900ac12653cfa564e88" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 45729 } } peers { permanent_uuid: "b70b290d8d4e444fae0462d65682bfa4" member_type: VOTER last_known_addr { host: "127.6.115.125" port: 35771 } attrs { promote: false } } peers { permanent_uuid: "e37585beb7a84c379041754b1aceb780" member_type: VOTER last_known_addr { host: "127.6.115.124" port: 41865 } attrs { promote: false } } }
I20251210 14:58:44.487713 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37585beb7a84c379041754b1aceb780 [sys.catalog]: This master's current role is: FOLLOWER
I20251210 14:58:44.496061 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.492600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:38623 (local address 127.0.0.1:42676)
1210 14:58:44.492935 (+ 335us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.492949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:44.493034 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:44.493353 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:44.493356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:44.493371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:44.493653 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.493662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.494792 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.494796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:44.495680 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.495688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.495785 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.495805 (+ 20us) client_negotiation.cc:769] Sending connection context
1210 14:58:44.495856 (+ 51us) client_negotiation.cc:240] Negotiation successful
1210 14:58:44.495922 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20251210 14:58:44.496803 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.492850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43469 (local address 127.0.0.1:60180)
1210 14:58:44.493113 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.493125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:44.493245 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:44.494222 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:44.494226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:44.494246 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:44.494485 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.494492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.495562 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.495566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:44.496465 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.496473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.496579 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.496594 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:58:44.496641 (+ 47us) client_negotiation.cc:240] Negotiation successful
1210 14:58:44.496689 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20251210 14:58:44.497239 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.493303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:45829 (local address 127.0.0.1:40594)
1210 14:58:44.493541 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.493554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:44.493630 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:44.493980 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:44.493983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:44.493999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:44.494249 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.494255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.495168 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.495172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:44.495919 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:44.495927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.496038 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.496055 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:58:44.496125 (+ 70us) client_negotiation.cc:240] Negotiation successful
1210 14:58:44.497137 (+ 1012us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20251210 14:58:44.496562 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.492738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.6.115.65:38623)
1210 14:58:44.492983 (+ 245us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:44.492988 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:44.493003 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:44.493166 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:44.493169 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:44.493232 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:44.493768 (+ 536us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:44.493881 (+ 113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.494674 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.495800 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.496320 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.496358 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:44.496406 (+ 48us) server_negotiation.cc:300] Negotiation successful
1210 14:58:44.496458 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20251210 14:58:44.497318 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.493579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.6.115.67:43469)
1210 14:58:44.493944 (+ 365us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:44.493948 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:44.493962 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:58:44.494009 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:44.494012 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:44.494062 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:44.494618 (+ 556us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:44.494707 (+ 89us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.495440 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.496594 (+ 1154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.497045 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.497081 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:44.497132 (+ 51us) server_negotiation.cc:300] Negotiation successful
1210 14:58:44.497195 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":72,"threads_started":1}
I20251210 14:58:44.497727 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.493454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.6.115.66:45829)
1210 14:58:44.493688 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:44.493694 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:44.493710 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:58:44.493760 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:44.493763 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:44.493809 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:44.493884 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:44.494382 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.495054 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.496312 (+ 1258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.496796 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.496835 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:44.496887 (+ 52us) server_negotiation.cc:300] Negotiation successful
1210 14:58:44.496971 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20251210 14:58:44.497789 14559 tablet_service.cc:1505] Processing CreateTablet for tablet fdaa59540a2b472f937dbfc198b10275 (DEFAULT_TABLE table=test-table [id=8b732195f6f548e2a504d79b5b66c228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:44.498075 14559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdaa59540a2b472f937dbfc198b10275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:44.497754 14287 tablet_service.cc:1505] Processing CreateTablet for tablet fdaa59540a2b472f937dbfc198b10275 (DEFAULT_TABLE table=test-table [id=8b732195f6f548e2a504d79b5b66c228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:44.498358 14287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdaa59540a2b472f937dbfc198b10275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:44.500039 14721 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Bootstrap starting.
I20251210 14:58:44.500375 14423 tablet_service.cc:1505] Processing CreateTablet for tablet fdaa59540a2b472f937dbfc198b10275 (DEFAULT_TABLE table=test-table [id=8b732195f6f548e2a504d79b5b66c228]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:44.500629 14423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdaa59540a2b472f937dbfc198b10275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:44.500974 14721 tablet_bootstrap.cc:654] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:44.501816 14722 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: Bootstrap starting.
I20251210 14:58:44.502213 14721 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: No bootstrap required, opened a new log
I20251210 14:58:44.502312 14721 ts_tablet_manager.cc:1403] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:44.502813 14724 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Bootstrap starting.
I20251210 14:58:44.502813 14721 raft_consensus.cc:359] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.502909 14722 tablet_bootstrap.cc:654] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:44.502941 14721 raft_consensus.cc:385] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:44.502982 14721 raft_consensus.cc:740] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27193df16bea4b3689ad1f635c99d1f4, State: Initialized, Role: FOLLOWER
I20251210 14:58:44.503145 14721 consensus_queue.cc:260] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.503553 14721 ts_tablet_manager.cc:1434] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251210 14:58:44.504566 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.503980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:44537)
1210 14:58:44.504422 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.504482 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":92,"threads_started":1}
I20251210 14:58:44.505383 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.504977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:49853)
1210 14:58:44.505232 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.505285 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20251210 14:58:44.505478 14724 tablet_bootstrap.cc:654] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:44.505970 14627 heartbeater.cc:499] Master 127.6.115.125:35771 was elected leader, sending a full tablet report...
I20251210 14:58:44.506799 14724 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: No bootstrap required, opened a new log
I20251210 14:58:44.506799 14722 tablet_bootstrap.cc:492] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: No bootstrap required, opened a new log
I20251210 14:58:44.506899 14724 ts_tablet_manager.cc:1403] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251210 14:58:44.506899 14722 ts_tablet_manager.cc:1403] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251210 14:58:44.507354 14724 raft_consensus.cc:359] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.507354 14722 raft_consensus.cc:359] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.507475 14722 raft_consensus.cc:385] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:44.507475 14724 raft_consensus.cc:385] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:44.507514 14722 raft_consensus.cc:740] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f17f9a97d6842679ebee2dfca09fd81, State: Initialized, Role: FOLLOWER
I20251210 14:58:44.507514 14724 raft_consensus.cc:740] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e554247114c24a86a457a3600f421282, State: Initialized, Role: FOLLOWER
I20251210 14:58:44.507665 14724 consensus_queue.cc:260] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.507665 14722 consensus_queue.cc:260] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.508003 14724 ts_tablet_manager.cc:1434] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:44.508003 14722 ts_tablet_manager.cc:1434] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:44.508603 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.508237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:53727)
1210 14:58:44.508466 (+ 229us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.508516 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20251210 14:58:44.509550 14725 raft_consensus.cc:493] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:44.509627 14725 raft_consensus.cc:515] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.509776 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.509530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:52547)
1210 14:58:44.509676 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.509721 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20251210 14:58:44.509974 14725 leader_election.cc:290] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623), e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:44.510432 14307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdaa59540a2b472f937dbfc198b10275" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f17f9a97d6842679ebee2dfca09fd81" is_pre_election: true
I20251210 14:58:44.510540 14307 raft_consensus.cc:2468] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 0.
I20251210 14:58:44.510576 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdaa59540a2b472f937dbfc198b10275" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554247114c24a86a457a3600f421282" is_pre_election: true
I20251210 14:58:44.510691 14443 raft_consensus.cc:2468] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 0.
I20251210 14:58:44.510949 14513 leader_election.cc:304] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27193df16bea4b3689ad1f635c99d1f4, e554247114c24a86a457a3600f421282; no voters:
I20251210 14:58:44.511243 14355 heartbeater.cc:499] Master 127.6.115.125:35771 was elected leader, sending a full tablet report...
I20251210 14:58:44.511607 14725 raft_consensus.cc:2804] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:44.511673 14725 raft_consensus.cc:493] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:44.511722 14725 raft_consensus.cc:3060] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:44.511852 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.509028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:40479)
1210 14:58:44.511705 (+ 2677us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.511755 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2599,"thread_start_us":88,"threads_started":1}
I20251210 14:58:44.512694 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.512261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:54539)
1210 14:58:44.512534 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.512591 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20251210 14:58:44.512681 14725 raft_consensus.cc:515] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.513053 14725 leader_election.cc:290] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623), e554247114c24a86a457a3600f421282 (127.6.115.66:45829)
I20251210 14:58:44.513197 14491 heartbeater.cc:499] Master 127.6.115.125:35771 was elected leader, sending a full tablet report...
I20251210 14:58:44.513389 14443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdaa59540a2b472f937dbfc198b10275" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554247114c24a86a457a3600f421282"
I20251210 14:58:44.513523 14443 raft_consensus.cc:3060] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:44.514353 14443 raft_consensus.cc:2468] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 1.
I20251210 14:58:44.514616 14513 leader_election.cc:304] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27193df16bea4b3689ad1f635c99d1f4, e554247114c24a86a457a3600f421282; no voters:
I20251210 14:58:44.514722 14307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdaa59540a2b472f937dbfc198b10275" candidate_uuid: "27193df16bea4b3689ad1f635c99d1f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f17f9a97d6842679ebee2dfca09fd81"
I20251210 14:58:44.514799 14725 raft_consensus.cc:2804] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:44.514832 14307 raft_consensus.cc:3060] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:44.514991 14725 raft_consensus.cc:697] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 LEADER]: Becoming Leader. State: Replica: 27193df16bea4b3689ad1f635c99d1f4, State: Running, Role: LEADER
I20251210 14:58:44.515136 14725 consensus_queue.cc:237] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } }
I20251210 14:58:44.515672 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.515436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:46075)
1210 14:58:44.515535 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.515584 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20251210 14:58:44.515838 14307 raft_consensus.cc:2468] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27193df16bea4b3689ad1f635c99d1f4 in term 1.
W20251210 14:58:44.515972 14356 tablet.cc:2378] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:44.517333 13970 catalog_manager.cc:5654] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "27193df16bea4b3689ad1f635c99d1f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27193df16bea4b3689ad1f635c99d1f4" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:44.529994 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.527865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.6.115.67:43469)
1210 14:58:44.527970 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:44.527974 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:44.527985 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:44.528020 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:44.528022 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:44.528058 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:44.528115 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:44.528401 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.528846 (+ 445us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:44.529497 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:44.529630 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:44.529668 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:44.529730 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:44.529786 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:44.529868 (+ 82us) server_negotiation.cc:300] Negotiation successful
1210 14:58:44.529904 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251210 14:58:44.531855 14443 raft_consensus.cc:1275] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Refusing update from remote peer 27193df16bea4b3689ad1f635c99d1f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:44.531855 14307 raft_consensus.cc:1275] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Refusing update from remote peer 27193df16bea4b3689ad1f635c99d1f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:44.532265 14725 consensus_queue.cc:1048] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f17f9a97d6842679ebee2dfca09fd81" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:44.532371 14735 consensus_queue.cc:1048] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e554247114c24a86a457a3600f421282" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:44.534073 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.533853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:35731)
1210 14:58:44.533967 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.534016 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251210 14:58:44.534392 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.533851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:39611)
1210 14:58:44.534274 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.534327 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
I20251210 14:58:44.536275 14738 mvcc.cc:204] Tried to move back new op lower bound from 7230991255678836736 to 7230991255615418368. Current Snapshot: MvccSnapshot[applied={T|T < 7230991255678836736}]
I20251210 14:58:44.536880 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.536607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:51477)
1210 14:58:44.536761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.536814 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20251210 14:58:44.538789 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.538293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.66:33287)
1210 14:58:44.538681 (+ 388us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.538728 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323}
I20251210 14:58:44.539541 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.538974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.67:39509)
1210 14:58:44.539427 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.539482 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388}
I20251210 14:58:44.540179 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.539944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:45729 (local address 127.6.115.65:39545)
1210 14:58:44.540073 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.540118 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20251210 14:58:44.542517 13969 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36530:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:44.542651 13969 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36530:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:44.546058 13969 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b70b290d8d4e444fae0462d65682bfa4: Sending DeleteTablet for 3 replicas of tablet fdaa59540a2b472f937dbfc198b10275
I20251210 14:58:44.546623 14287 tablet_service.cc:1552] Processing DeleteTablet for tablet fdaa59540a2b472f937dbfc198b10275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:44 UTC) from {username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:42676
I20251210 14:58:44.546706 14559 tablet_service.cc:1552] Processing DeleteTablet for tablet fdaa59540a2b472f937dbfc198b10275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:44 UTC) from {username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:60180
I20251210 14:58:44.546984 14747 tablet_replica.cc:333] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81: stopping tablet replica
I20251210 14:58:44.547049 14748 tablet_replica.cc:333] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: stopping tablet replica
I20251210 14:58:44.547122 14747 raft_consensus.cc:2243] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:44.547164 14748 raft_consensus.cc:2243] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:44.547281 14747 raft_consensus.cc:2272] T fdaa59540a2b472f937dbfc198b10275 P 0f17f9a97d6842679ebee2dfca09fd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:44.547338 14748 raft_consensus.cc:2272] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:44.547433 14423 tablet_service.cc:1552] Processing DeleteTablet for tablet fdaa59540a2b472f937dbfc198b10275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:44 UTC) from {username='oryx', principal='oryx/127.6.115.125@KRBTEST.COM'} at 127.0.0.1:40594
I20251210 14:58:44.548035 14748 ts_tablet_manager.cc:1916] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:44.548101 14749 tablet_replica.cc:333] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: stopping tablet replica
I20251210 14:58:44.548236 14749 raft_consensus.cc:2243] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:44.548411 14749 raft_consensus.cc:2272] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:44.548552 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 14224
I20251210 14:58:44.550575 14749 ts_tablet_manager.cc:1916] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:44.552273 14748 ts_tablet_manager.cc:1929] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:44.552356 14748 log.cc:1199] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/fdaa59540a2b472f937dbfc198b10275
I20251210 14:58:44.552454 14749 ts_tablet_manager.cc:1929] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:44.552534 14749 log.cc:1199] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/fdaa59540a2b472f937dbfc198b10275
I20251210 14:58:44.552654 14748 ts_tablet_manager.cc:1950] T fdaa59540a2b472f937dbfc198b10275 P 27193df16bea4b3689ad1f635c99d1f4: Deleting consensus metadata
I20251210 14:58:44.552839 14749 ts_tablet_manager.cc:1950] T fdaa59540a2b472f937dbfc198b10275 P e554247114c24a86a457a3600f421282: Deleting consensus metadata
I20251210 14:58:44.553344 13955 catalog_manager.cc:4985] TS 27193df16bea4b3689ad1f635c99d1f4 (127.6.115.67:43469): tablet fdaa59540a2b472f937dbfc198b10275 (table test-table [id=8b732195f6f548e2a504d79b5b66c228]) successfully deleted
I20251210 14:58:44.553438 13956 catalog_manager.cc:4985] TS e554247114c24a86a457a3600f421282 (127.6.115.66:45829): tablet fdaa59540a2b472f937dbfc198b10275 (table test-table [id=8b732195f6f548e2a504d79b5b66c228]) successfully deleted
I20251210 14:58:44.555903 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 14360
I20251210 14:58:44.556655 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:44.555945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:38623 (local address 127.0.0.1:42678)
1210 14:58:44.556503 (+ 558us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:44.556563 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:38623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20251210 14:58:44.556810 13956 catalog_manager.cc:4712] TS 0f17f9a97d6842679ebee2dfca09fd81 (127.6.115.65:38623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdaa59540a2b472f937dbfc198b10275: Network error: Client connection negotiation failed: client connection to 127.6.115.65:38623: connect: Connection refused (error 111)
I20251210 14:58:44.562734 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 14496
I20251210 14:58:44.569280 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13831
I20251210 14:58:44.574896 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 13943
2025-12-10T14:58:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9155 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:44 dist-test-slave-nb51 krb5kdc[14752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:44 dist-test-slave-nb51 krb5kdc[14752](info): set up 2 sockets
Dec 10 14:58:44 dist-test-slave-nb51 krb5kdc[14752](info): commencing operation
krb5kdc: starting...
W20251210 14:58:46.550295 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:46 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:46.682592 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43451
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:46.786293 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:46.786505 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:46.786586 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:46.789767 14768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:46.789821 14768 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251210 14:58:46.789848 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:46.789891 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:46.789923 14768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:46.789955 14768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:46.794071 14768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:43451
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:46.795084 14768 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:46.795943 14768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:46.801499 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:46.801530 14773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:46.801499 14774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:46.801640 14768 server_base.cc:1047] running on GCE node
I20251210 14:58:46.802300 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:46.803159 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:46.804328 14768 hybrid_clock.cc:648] HybridClock initialized: now 1765378726804318 us; error 25 us; skew 500 ppm
Dec 10 14:58:46 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378726, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:46.806882 14768 init.cc:377] Logged in from keytab as oryx/127.6.115.126@KRBTEST.COM (short username oryx)
I20251210 14:58:46.807861 14768 webserver.cc:492] Webserver started at http://127.6.115.126:36967/ using document root <none> and password file <none>
I20251210 14:58:46.808362 14768 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:46.808411 14768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:46.808578 14768 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:46.810250 14768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "ca2f14f7a9144026af8265ac2ee887d6"
format_stamp: "Formatted at 2025-12-10 14:58:46 on dist-test-slave-nb51"
server_key: "1dcb90b6e0c769159d5a17ddc88a56b9"
server_key_iv: "583955712832b6d1ee760ce3c2c234f7"
server_key_version: "encryptionkey@0"
I20251210 14:58:46.810695 14768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "ca2f14f7a9144026af8265ac2ee887d6"
format_stamp: "Formatted at 2025-12-10 14:58:46 on dist-test-slave-nb51"
server_key: "1dcb90b6e0c769159d5a17ddc88a56b9"
server_key_iv: "583955712832b6d1ee760ce3c2c234f7"
server_key_version: "encryptionkey@0"
I20251210 14:58:46.813795 14768 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:46.815820 14783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:46.816766 14768 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:46.816869 14768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "ca2f14f7a9144026af8265ac2ee887d6"
format_stamp: "Formatted at 2025-12-10 14:58:46 on dist-test-slave-nb51"
server_key: "1dcb90b6e0c769159d5a17ddc88a56b9"
server_key_iv: "583955712832b6d1ee760ce3c2c234f7"
server_key_version: "encryptionkey@0"
I20251210 14:58:46.817081 14768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:46.827064 14768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:46.833750 14768 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:46.833978 14768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:46.841578 14768 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:43451
I20251210 14:58:46.841832 14845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:43451 every 8 connection(s)
I20251210 14:58:46.842542 14768 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:46.845279 14846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:46.847952 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 14768
I20251210 14:58:46.848064 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:46.848295 6605 external_mini_cluster.cc:1468] Setting key 37e1ba9ccaed433fb7703df7e2a07c93
I20251210 14:58:46.850692 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Bootstrap starting.
I20251210 14:58:46.853207 14846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:46.853996 14846 log.cc:826] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:46.855782 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: No bootstrap required, opened a new log
Dec 10 14:58:46 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:46.858632 14846 raft_consensus.cc:359] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } }
I20251210 14:58:46.858803 14846 raft_consensus.cc:385] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:46.858899 14846 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2f14f7a9144026af8265ac2ee887d6, State: Initialized, Role: FOLLOWER
I20251210 14:58:46.859326 14846 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } }
I20251210 14:58:46.859453 14846 raft_consensus.cc:399] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:46.859519 14846 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:46.859629 14846 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:46.860401 14846 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } }
I20251210 14:58:46.860773 14846 leader_election.cc:304] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2f14f7a9144026af8265ac2ee887d6; no voters:
I20251210 14:58:46.861068 14846 leader_election.cc:290] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:46.861289 14851 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:46.861486 14851 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [term 1 LEADER]: Becoming Leader. State: Replica: ca2f14f7a9144026af8265ac2ee887d6, State: Running, Role: LEADER
I20251210 14:58:46.861804 14851 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } }
I20251210 14:58:46.862138 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:46.849562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.6.115.126:43451)
1210 14:58:46.849988 (+ 426us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:46.850000 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:46.850041 (+ 41us) server_negotiation.cc:408] Connection header received
1210 14:58:46.850702 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:46.850727 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:46.851082 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:46.851423 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:46.852199 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:46.853318 (+ 1119us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:46.854045 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:46.854325 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:46.856638 (+ 2313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:46.856655 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:46.856665 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:46.856695 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:46.859260 (+ 2565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:46.859728 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:46.859737 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:46.859741 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:46.859811 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:46.860124 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:46.860129 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:46.860131 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:46.860440 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:46.860638 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:46.860949 (+ 311us) server_negotiation.cc:300] Negotiation successful
1210 14:58:46.861293 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20251210 14:58:46.862226 14846 sys_catalog.cc:565] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:46.862952 14851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2f14f7a9144026af8265ac2ee887d6. Latest consensus state: current_term: 1 leader_uuid: "ca2f14f7a9144026af8265ac2ee887d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } } }
I20251210 14:58:46.863057 14851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:46.864252 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2f14f7a9144026af8265ac2ee887d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2f14f7a9144026af8265ac2ee887d6" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 43451 } } }
I20251210 14:58:46.864385 14859 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:46.864428 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6 [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:46.867305 14859 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:46.871711 14859 catalog_manager.cc:1357] Generated new cluster ID: b8505d02ed53495f9c151bcae0ec7292
I20251210 14:58:46.871771 14859 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:46.876242 14859 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:46.877319 14859 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:46.891844 14859 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Generated new TSK 0
I20251210 14:58:46.892517 14859 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:46.944330 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:43451
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:47.090770 14874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:47.091128 14874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:47.091223 14874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:47.095947 14874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:47.096040 14874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:47.096129 14874 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:58:47.096217 14874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:47.101874 14874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:47.103266 14874 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:47.104334 14874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:47.110625 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.110627 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.110627 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:47.110886 14874 server_base.cc:1047] running on GCE node
I20251210 14:58:47.111202 14874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:47.111724 14874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:47.112864 14874 hybrid_clock.cc:648] HybridClock initialized: now 1765378727112854 us; error 43 us; skew 500 ppm
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:47.115410 14874 init.cc:377] Logged in from keytab as oryx/127.6.115.65@KRBTEST.COM (short username oryx)
I20251210 14:58:47.116369 14874 webserver.cc:492] Webserver started at http://127.6.115.65:44249/ using document root <none> and password file <none>
I20251210 14:58:47.116904 14874 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:47.116957 14874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:47.117139 14874 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:47.118873 14874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "29ff8298ac0d407386639b3834a51435"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "d198854cd51e0a576d0bb9f41f81e93e"
server_key_iv: "5e67123ab28f124cd623e3fba0094dac"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.119339 14874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "29ff8298ac0d407386639b3834a51435"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "d198854cd51e0a576d0bb9f41f81e93e"
server_key_iv: "5e67123ab28f124cd623e3fba0094dac"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.122558 14874 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:47.124614 14889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.125757 14874 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:47.125876 14874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "29ff8298ac0d407386639b3834a51435"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "d198854cd51e0a576d0bb9f41f81e93e"
server_key_iv: "5e67123ab28f124cd623e3fba0094dac"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.125972 14874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:47.137177 14874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:47.140373 14874 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:47.140545 14874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:47.142032 14874 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:47.142083 14874 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.142139 14874 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:47.142175 14874 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:47.152905 14874 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:41841
I20251210 14:58:47.153059 15004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:41841 every 8 connection(s)
I20251210 14:58:47.154124 14874 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:47.156077 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.142676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:37737 (local address 127.6.115.126:43451)
1210 14:58:47.142837 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.142844 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.143698 (+ 854us) server_negotiation.cc:408] Connection header received
1210 14:58:47.144689 (+ 991us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.144693 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.144764 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.145413 (+ 649us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.147590 (+ 2177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.148388 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.149388 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.149604 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.152033 (+ 2429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.152055 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.152058 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.152088 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.154051 (+ 1963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.154628 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.154632 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.154634 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.154686 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.155074 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.155079 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.155084 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.155282 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.155373 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.155837 (+ 464us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.155952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251210 14:58:47.156642 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.143046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.65:37737)
1210 14:58:47.143529 (+ 483us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.143570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.144434 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.145996 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.146010 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.146606 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.147388 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.147409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.148549 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.148554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.149233 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.149246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.149514 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.150143 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.150178 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.151881 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.154207 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.154214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.154227 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.154490 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.154804 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.154808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.154810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.154954 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.155412 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.155417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.155607 (+ 190us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.155783 (+ 176us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.155984 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20251210 14:58:47.157864 15005 heartbeater.cc:344] Connected to a master server at 127.6.115.126:43451
I20251210 14:58:47.158089 15005 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:47.158686 15005 heartbeater.cc:507] Master 127.6.115.126:43451 requested a full tablet report, sending...
I20251210 14:58:47.160158 14800 ts_manager.cc:194] Registered new tserver with Master: 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841)
I20251210 14:58:47.160730 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 14874
I20251210 14:58:47.160900 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:47.161177 6605 external_mini_cluster.cc:1468] Setting key fbb2af66ff34207d472193de35abc314
I20251210 14:58:47.161788 14800 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:37737
I20251210 14:58:47.167284 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.160195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:39569 (local address 127.6.115.126:43451)
1210 14:58:47.160336 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.160340 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.160526 (+ 186us) server_negotiation.cc:408] Connection header received
1210 14:58:47.160626 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.160630 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.160689 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.160790 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.161483 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.162167 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.162921 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.163112 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.164319 (+ 1207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.164332 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.164334 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.164355 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.165795 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.166294 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.166297 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.166298 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.166337 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.166594 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.166596 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.166597 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.166707 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.166772 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.167093 (+ 321us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.167181 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251210 14:58:47.167578 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.160065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.65:39569)
1210 14:58:47.160426 (+ 361us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.160442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.160538 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.160895 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.160898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.161080 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.161352 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.161383 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.162315 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.162320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.162782 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.162793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.162924 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.163685 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.163708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.164123 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.165920 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.165927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.165932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.166179 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.166433 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.166437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.166441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.166502 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.166800 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.166806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.166920 (+ 114us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.167062 (+ 142us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.167217 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:47.208761 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:43451
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:47.309007 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:47.309201 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:47.309260 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:47.312438 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:47.312497 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:47.312556 15016 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:58:47.312633 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:47.316774 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:47.317866 15016 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:47.318666 15016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:47.325459 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.325465 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.325459 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:47.325690 15016 server_base.cc:1047] running on GCE node
I20251210 14:58:47.326159 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:47.326628 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:47.327791 15016 hybrid_clock.cc:648] HybridClock initialized: now 1765378727327767 us; error 38 us; skew 500 ppm
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:47.330384 15016 init.cc:377] Logged in from keytab as oryx/127.6.115.66@KRBTEST.COM (short username oryx)
I20251210 14:58:47.331357 15016 webserver.cc:492] Webserver started at http://127.6.115.66:42273/ using document root <none> and password file <none>
I20251210 14:58:47.331883 15016 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:47.331933 15016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:47.332103 15016 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:47.333757 15016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "4bba6fe07ad648dcbcdefe5344066137"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "860d94039daa979b80ff827d52b3bc5a"
server_key_iv: "ac108e641a74a75ff9b09d61ce9babcd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.334259 15016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "4bba6fe07ad648dcbcdefe5344066137"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "860d94039daa979b80ff827d52b3bc5a"
server_key_iv: "ac108e641a74a75ff9b09d61ce9babcd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.337317 15016 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:58:47.339344 15031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.340310 15016 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251210 14:58:47.340409 15016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "4bba6fe07ad648dcbcdefe5344066137"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "860d94039daa979b80ff827d52b3bc5a"
server_key_iv: "ac108e641a74a75ff9b09d61ce9babcd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.340494 15016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:47.354324 15016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:47.357154 15016 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:47.357326 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:47.358700 15016 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:47.358753 15016 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.358812 15016 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:47.358852 15016 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:47.369546 15016 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:43167
I20251210 14:58:47.369599 15146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:43167 every 8 connection(s)
I20251210 14:58:47.370644 15016 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:47.371433 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.359359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40495 (local address 127.6.115.126:43451)
1210 14:58:47.359479 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.359483 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.360299 (+ 816us) server_negotiation.cc:408] Connection header received
1210 14:58:47.361100 (+ 801us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.361104 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.361167 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.361249 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.362917 (+ 1668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.363666 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.364652 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.364874 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.367672 (+ 2798us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.367693 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.367698 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.367730 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.369517 (+ 1787us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.370042 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.370045 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.370046 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.370100 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.370395 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.370397 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.370398 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.370527 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.370594 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.371166 (+ 572us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.371277 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:58:47.372033 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.359715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.66:40495)
1210 14:58:47.360166 (+ 451us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.360199 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.360902 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.361483 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.361491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.362004 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.362743 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.362758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.363814 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.363820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.364500 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.364536 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.364764 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.365589 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.365625 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.367512 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.369654 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.369660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.369670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.369941 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.370206 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.370209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.370211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.370307 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.370624 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.370632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.370893 (+ 261us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.371130 (+ 237us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.371370 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20251210 14:58:47.373049 15147 heartbeater.cc:344] Connected to a master server at 127.6.115.126:43451
I20251210 14:58:47.373190 15147 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:47.373773 15147 heartbeater.cc:507] Master 127.6.115.126:43451 requested a full tablet report, sending...
I20251210 14:58:47.373796 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15016
I20251210 14:58:47.373900 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:47.374181 6605 external_mini_cluster.cc:1468] Setting key ac27be29b780bdb1aad5a85778999670
I20251210 14:58:47.374804 14800 ts_manager.cc:194] Registered new tserver with Master: 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167)
I20251210 14:58:47.375481 14800 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:40495
I20251210 14:58:47.382169 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.375237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.66:54887)
1210 14:58:47.375551 (+ 314us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.375565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.375662 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.375899 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.375902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.376116 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.376386 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.376396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.377389 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.377394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.378025 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.378039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.378150 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.378647 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.378659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.378913 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.380881 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.380883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.380885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.381034 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.381288 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.381290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.381292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.381328 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.381803 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.381806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.381873 (+ 67us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.381951 (+ 78us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.382036 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20251210 14:58:47.382333 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.375301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:54887 (local address 127.6.115.126:43451)
1210 14:58:47.375415 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.375419 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.375651 (+ 232us) server_negotiation.cc:408] Connection header received
1210 14:58:47.375739 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.375742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.375787 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.375863 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.376519 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.377224 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.378145 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.378273 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.379039 (+ 766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.379058 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.379064 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.379095 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.380758 (+ 1663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.381136 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.381142 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.381146 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.381199 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.381441 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.381446 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.381447 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.381642 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.381803 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.382031 (+ 228us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.382156 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab.
I20251210 14:58:47.420468 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:43451
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:47.523917 15158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:47.524113 15158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:47.524207 15158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:47.527458 15158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:47.527520 15158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:47.527614 15158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251210 14:58:47.527691 15158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:47.531759 15158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/krb5kdc/oryx.keytab
--principal=oryx/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.115.126:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:47.532709 15158 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:47.533471 15158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:47.539515 15164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.539523 15166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:47.539614 15163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:47.539855 15158 server_base.cc:1047] running on GCE node
I20251210 14:58:47.540203 15158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:47.540683 15158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:47.541828 15158 hybrid_clock.cc:648] HybridClock initialized: now 1765378727541825 us; error 35 us; skew 500 ppm
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:47.544405 15158 init.cc:377] Logged in from keytab as oryx/127.6.115.67@KRBTEST.COM (short username oryx)
I20251210 14:58:47.545468 15158 webserver.cc:492] Webserver started at http://127.6.115.67:35623/ using document root <none> and password file <none>
I20251210 14:58:47.546136 15158 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:47.546202 15158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:47.546401 15158 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:47.548046 15158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "21d05794bfb44748b737d4086d0f4d39"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "85b03a43264f83bcb50079f21f7fa370"
server_key_iv: "c7b80e94910a1a1166ea963fb14284fd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.548497 15158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "21d05794bfb44748b737d4086d0f4d39"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "85b03a43264f83bcb50079f21f7fa370"
server_key_iv: "c7b80e94910a1a1166ea963fb14284fd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.551707 15158 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251210 14:58:47.553821 15173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.554756 15158 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251210 14:58:47.554864 15158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "21d05794bfb44748b737d4086d0f4d39"
format_stamp: "Formatted at 2025-12-10 14:58:47 on dist-test-slave-nb51"
server_key: "85b03a43264f83bcb50079f21f7fa370"
server_key_iv: "c7b80e94910a1a1166ea963fb14284fd"
server_key_version: "encryptionkey@0"
I20251210 14:58:47.554948 15158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:47.572443 15158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:47.575090 15158 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:47.575273 15158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:47.576785 15158 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:47.576848 15158 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:47.576900 15158 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:47.576934 15158 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:47.588806 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.577252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:50369 (local address 127.6.115.126:43451)
1210 14:58:47.577403 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.577407 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.578165 (+ 758us) server_negotiation.cc:408] Connection header received
1210 14:58:47.578999 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.579003 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.579082 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.579157 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.580702 (+ 1545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.581505 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.582200 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.582376 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.584887 (+ 2511us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.584914 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.584920 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.584951 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.586764 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.587422 (+ 658us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.587427 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.587430 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.587478 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.587849 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.587852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.587853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.587993 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.588083 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.588539 (+ 456us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.588648 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:58:47.588892 15158 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:40725
I20251210 14:58:47.588943 15288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:40725 every 8 connection(s)
I20251210 14:58:47.589542 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.577586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.67:50369)
1210 14:58:47.578034 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.578065 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.578772 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.579338 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.579348 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.579758 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.580525 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.580544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.581608 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.581615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.582077 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.582086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.582287 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.583038 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.583072 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.584698 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.586874 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.586883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.586895 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.587268 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.587605 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.587608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.587609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.587697 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.588106 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.588112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.588315 (+ 203us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.588487 (+ 172us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.588717 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20251210 14:58:47.590013 15158 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:47.591279 15289 heartbeater.cc:344] Connected to a master server at 127.6.115.126:43451
I20251210 14:58:47.591434 15289 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:47.591908 15289 heartbeater.cc:507] Master 127.6.115.126:43451 requested a full tablet report, sending...
I20251210 14:58:47.592891 14800 ts_manager.cc:194] Registered new tserver with Master: 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:47.593693 14800 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:50369
I20251210 14:58:47.595934 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15158
I20251210 14:58:47.596089 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:47.596370 6605 external_mini_cluster.cc:1468] Setting key af9a10690c65a9969f2a53d83555895a
I20251210 14:58:47.598845 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:58:47.599392 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.592741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.6.115.67:54069)
1210 14:58:47.593063 (+ 322us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.593080 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.593182 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.593430 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.593434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.593616 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.593885 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.593896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.594756 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.594759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.595170 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.595178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.595268 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.595764 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.595783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.596178 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.598085 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.598089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.598092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.598288 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.598541 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.598545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.598549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.598606 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.598988 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.598992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.599086 (+ 94us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.599179 (+ 93us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.599276 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
I20251210 14:58:47.599498 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.592786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:54069 (local address 127.6.115.126:43451)
1210 14:58:47.592903 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.592907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.593155 (+ 248us) server_negotiation.cc:408] Connection header received
1210 14:58:47.593266 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.593269 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.593306 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.593406 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.594010 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.594625 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.595294 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.595467 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.596290 (+ 823us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.596303 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.596305 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.596326 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.597953 (+ 1627us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.598390 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.598394 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.598396 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.598448 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.598699 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.598701 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.598703 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.598878 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.598978 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.599243 (+ 265us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.599359 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
W20251210 14:58:47.619519 14849 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:47.614249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.6.115.126:43451)
1210 14:58:47.614369 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.614372 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.614383 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:47.614416 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.614418 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.614456 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.614524 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.615327 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.615817 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.616497 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.616631 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.619394 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58182: BlockingRecv error: recv got EOF from 127.0.0.1:58182 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:47.627671 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.620855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.6.115.126:43451)
1210 14:58:47.620959 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.620963 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.621056 (+ 93us) server_negotiation.cc:408] Connection header received
1210 14:58:47.621151 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.621153 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.621187 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.621251 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.621976 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.622425 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.622996 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.623119 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.624999 (+ 1880us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.625014 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.625017 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.625039 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.626541 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.626912 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.626915 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.626916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.626951 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.627126 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.627127 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.627129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.627247 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.627340 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.627479 (+ 139us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.627555 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251210 14:58:47.630283 14800 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:47.632544 14800 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:47.645064 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.640343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:41841 (local address 127.0.0.1:39600)
1210 14:58:47.640671 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.640730 (+ 59us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.640864 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.641839 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.641845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.641869 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.642187 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.642197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643645 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.643651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.644545 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.644555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.644657 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.644705 (+ 48us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.644796 (+ 91us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.644887 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20251210 14:58:47.645067 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.640863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:43167 (local address 127.0.0.1:39796)
1210 14:58:47.641868 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.641881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.641964 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.642549 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.642552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.642566 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.642780 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.642794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643721 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.643725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.644641 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.644651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.644750 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.644768 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.644823 (+ 55us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.644888 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":94,"threads_started":1}
I20251210 14:58:47.645242 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.640526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.0.0.1:52340)
1210 14:58:47.641452 (+ 926us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.641467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.641546 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.642305 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.642308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.642331 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.642604 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.642614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643923 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.643926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.644831 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.644839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.644933 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.644946 (+ 13us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.644995 (+ 49us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.645053 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":95,"threads_started":1}
I20251210 14:58:47.645946 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.641252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.6.115.66:43167)
1210 14:58:47.641698 (+ 446us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.641707 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.642060 (+ 353us) server_negotiation.cc:408] Connection header received
1210 14:58:47.642131 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.642135 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.642270 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.642400 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.642923 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643584 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.644876 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.645450 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.645577 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.645644 (+ 67us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.645739 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":120,"threads_started":1}
I20251210 14:58:47.646091 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.641096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.6.115.67:40725)
1210 14:58:47.641905 (+ 809us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.641911 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.641931 (+ 20us) server_negotiation.cc:408] Connection header received
1210 14:58:47.642008 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.642014 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.642159 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.642305 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.642734 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643815 (+ 1081us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.645137 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.645689 (+ 552us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.645788 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.645848 (+ 60us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.645935 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":92,"threads_started":1}
I20251210 14:58:47.646816 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.640978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.6.115.65:41841)
1210 14:58:47.641283 (+ 305us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.641290 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.641315 (+ 25us) server_negotiation.cc:408] Connection header received
1210 14:58:47.641442 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.641447 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.641629 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.642317 (+ 688us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.642468 (+ 151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.643503 (+ 1035us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.646007 (+ 2504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.646485 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.646554 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.646602 (+ 48us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.646684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20251210 14:58:47.648407 15223 tablet_service.cc:1505] Processing CreateTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c (DEFAULT_TABLE table=test-table [id=41c00058a2f34c5190ee4fefd049aaf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.648507 15081 tablet_service.cc:1505] Processing CreateTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c (DEFAULT_TABLE table=test-table [id=41c00058a2f34c5190ee4fefd049aaf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.648856 14939 tablet_service.cc:1505] Processing CreateTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c (DEFAULT_TABLE table=test-table [id=41c00058a2f34c5190ee4fefd049aaf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.649232 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a602767b4a43eda51a53b5f4ebca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.649348 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a602767b4a43eda51a53b5f4ebca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.649580 14939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a602767b4a43eda51a53b5f4ebca4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.654454 15317 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Bootstrap starting.
I20251210 14:58:47.655622 15319 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Bootstrap starting.
I20251210 14:58:47.655622 15318 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Bootstrap starting.
I20251210 14:58:47.656507 15317 tablet_bootstrap.cc:654] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.657624 15319 tablet_bootstrap.cc:654] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.657624 15318 tablet_bootstrap.cc:654] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.657671 15317 log.cc:826] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:47.658466 15318 log.cc:826] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:47.658921 15319 log.cc:826] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:47.659724 15317 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: No bootstrap required, opened a new log
I20251210 14:58:47.659931 15317 ts_tablet_manager.cc:1403] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251210 14:58:47.660658 15319 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: No bootstrap required, opened a new log
I20251210 14:58:47.660840 15319 ts_tablet_manager.cc:1403] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:58:47.662304 15318 tablet_bootstrap.cc:492] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: No bootstrap required, opened a new log
I20251210 14:58:47.662513 15318 ts_tablet_manager.cc:1403] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251210 14:58:47.663305 15317 raft_consensus.cc:359] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.663582 15317 raft_consensus.cc:385] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.663676 15317 raft_consensus.cc:740] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.664111 15319 raft_consensus.cc:359] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.664180 15317 consensus_queue.cc:260] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.664315 15319 raft_consensus.cc:385] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.664407 15319 raft_consensus.cc:740] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.664865 15289 heartbeater.cc:499] Master 127.6.115.126:43451 was elected leader, sending a full tablet report...
I20251210 14:58:47.664857 15319 consensus_queue.cc:260] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.665172 15317 ts_tablet_manager.cc:1434] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251210 14:58:47.665555 15318 raft_consensus.cc:359] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.665688 15319 ts_tablet_manager.cc:1434] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251210 14:58:47.665791 15318 raft_consensus.cc:385] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.665824 15147 heartbeater.cc:499] Master 127.6.115.126:43451 was elected leader, sending a full tablet report...
I20251210 14:58:47.665886 15318 raft_consensus.cc:740] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.666229 15318 consensus_queue.cc:260] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.666869 15005 heartbeater.cc:499] Master 127.6.115.126:43451 was elected leader, sending a full tablet report...
I20251210 14:58:47.667140 15318 ts_tablet_manager.cc:1434] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:58:47.701964 15323 raft_consensus.cc:493] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:47.702163 15323 raft_consensus.cc:515] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.703317 15323 leader_election.cc:290] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167)
I20251210 14:58:47.706195 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.703565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:43167 (local address 127.6.115.67:48207)
1210 14:58:47.703704 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.703720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.703823 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.704108 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.704112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.704133 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.704329 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.704335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.705181 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.705184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.705793 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.705807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.705919 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.705942 (+ 23us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.705996 (+ 54us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.706068 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251210 14:58:47.706674 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.703703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:48207 (local address 127.6.115.66:43167)
1210 14:58:47.703851 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.703858 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.703875 (+ 17us) server_negotiation.cc:408] Connection header received
1210 14:58:47.703931 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.703934 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.703979 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.704048 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.704457 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.705052 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.705922 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.706396 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.706444 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.706505 (+ 61us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.706560 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251210 14:58:47.707070 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.703565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:41841 (local address 127.6.115.67:41293)
1210 14:58:47.704006 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.704017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.704119 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.704367 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.704370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.704385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.704611 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.704617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.705393 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.705396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.706025 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.706042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.706727 (+ 685us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.706742 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.706868 (+ 126us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.706925 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":167,"thread_start_us":117,"threads_started":1}
I20251210 14:58:47.707116 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.703674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:41293 (local address 127.6.115.65:41841)
1210 14:58:47.703854 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.703861 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.704105 (+ 244us) server_negotiation.cc:408] Connection header received
1210 14:58:47.704234 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.704238 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.704280 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.704345 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.704726 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.705256 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.706189 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.706630 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.706669 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.706934 (+ 265us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.707001 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251210 14:58:47.707376 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30a602767b4a43eda51a53b5f4ebca4c" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137" is_pre_election: true
I20251210 14:58:47.707667 15101 raft_consensus.cc:2468] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 0.
I20251210 14:58:47.707653 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30a602767b4a43eda51a53b5f4ebca4c" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435" is_pre_election: true
I20251210 14:58:47.707892 14959 raft_consensus.cc:2468] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 0.
I20251210 14:58:47.708143 15175 leader_election.cc:304] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:47.708364 15323 raft_consensus.cc:2804] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:47.708432 15323 raft_consensus.cc:493] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:47.708462 15323 raft_consensus.cc:3060] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.709409 15323 raft_consensus.cc:515] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.709764 15323 leader_election.cc:290] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 election: Requested vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167)
I20251210 14:58:47.710165 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30a602767b4a43eda51a53b5f4ebca4c" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137"
I20251210 14:58:47.710160 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30a602767b4a43eda51a53b5f4ebca4c" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435"
I20251210 14:58:47.710274 15101 raft_consensus.cc:3060] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.710289 14959 raft_consensus.cc:3060] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.711222 15101 raft_consensus.cc:2468] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 1.
I20251210 14:58:47.711297 14959 raft_consensus.cc:2468] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 1.
I20251210 14:58:47.711510 15175 leader_election.cc:304] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:47.711675 15323 raft_consensus.cc:2804] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:47.711872 15323 raft_consensus.cc:697] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 1 LEADER]: Becoming Leader. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Running, Role: LEADER
I20251210 14:58:47.712172 15323 consensus_queue.cc:237] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } }
I20251210 14:58:47.715247 14799 catalog_manager.cc:5654] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:47.729259 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.726392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.6.115.67:40725)
1210 14:58:47.726542 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.726546 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.726560 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:58:47.726676 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.726678 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.726728 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.726812 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:47.727154 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.727651 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.728288 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.728415 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.728459 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:47.728881 (+ 422us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:47.728965 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.729047 (+ 82us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.729088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251210 14:58:47.734335 14959 raft_consensus.cc:1275] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Refusing update from remote peer 21d05794bfb44748b737d4086d0f4d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:47.734334 15101 raft_consensus.cc:1275] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Refusing update from remote peer 21d05794bfb44748b737d4086d0f4d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:47.734906 15327 consensus_queue.cc:1048] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:47.735044 15323 consensus_queue.cc:1048] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:47.742688 15332 mvcc.cc:204] Tried to move back new op lower bound from 7230991268793823232 to 7230991268711772160. Current Snapshot: MvccSnapshot[applied={T|T < 7230991268793823232}]
I20251210 14:58:47.750061 14799 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:47.750232 14799 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:47.752609 14799 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Sending DeleteTablet for 3 replicas of tablet 30a602767b4a43eda51a53b5f4ebca4c
I20251210 14:58:47.753238 14939 tablet_service.cc:1552] Processing DeleteTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:47 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39600
I20251210 14:58:47.753330 15081 tablet_service.cc:1552] Processing DeleteTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:47 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39796
I20251210 14:58:47.753321 15223 tablet_service.cc:1552] Processing DeleteTablet for tablet 30a602767b4a43eda51a53b5f4ebca4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:47 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:52340
I20251210 14:58:47.753854 15340 tablet_replica.cc:333] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: stopping tablet replica
I20251210 14:58:47.754031 15341 tablet_replica.cc:333] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: stopping tablet replica
I20251210 14:58:47.754139 15340 raft_consensus.cc:2243] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:47.754376 15341 raft_consensus.cc:2243] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:47.754437 15340 raft_consensus.cc:2272] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:47.754467 14799 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251210 14:58:47.754880 15341 raft_consensus.cc:2272] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251210 14:58:47.754886 14799 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:47.755815 15340 ts_tablet_manager.cc:1916] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:47.756549 15341 ts_tablet_manager.cc:1916] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:47.756891 15342 tablet_replica.cc:333] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: stopping tablet replica
I20251210 14:58:47.757118 15342 raft_consensus.cc:2243] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:47.757483 15342 raft_consensus.cc:2272] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:47.759061 15340 ts_tablet_manager.cc:1929] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:47.759159 15340 log.cc:1199] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/30a602767b4a43eda51a53b5f4ebca4c
I20251210 14:58:47.759732 14939 tablet_service.cc:1505] Processing CreateTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 (DEFAULT_TABLE table=test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.759974 14939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa4e86b8784f159d6aa8d5100f9802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.759956 15081 tablet_service.cc:1505] Processing CreateTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 (DEFAULT_TABLE table=test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.760063 15223 tablet_service.cc:1505] Processing CreateTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 (DEFAULT_TABLE table=test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:47.760205 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa4e86b8784f159d6aa8d5100f9802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.760295 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa4e86b8784f159d6aa8d5100f9802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.761689 15341 ts_tablet_manager.cc:1929] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:47.761775 15341 log.cc:1199] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/30a602767b4a43eda51a53b5f4ebca4c
I20251210 14:58:47.762070 15341 ts_tablet_manager.cc:1950] T 30a602767b4a43eda51a53b5f4ebca4c P 4bba6fe07ad648dcbcdefe5344066137: Deleting consensus metadata
I20251210 14:58:47.762528 15319 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Bootstrap starting.
I20251210 14:58:47.762532 15317 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Bootstrap starting.
I20251210 14:58:47.763479 15319 tablet_bootstrap.cc:654] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.763489 15342 ts_tablet_manager.cc:1916] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:47.763617 15317 tablet_bootstrap.cc:654] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.764169 15318 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Bootstrap starting.
I20251210 14:58:47.764382 15340 ts_tablet_manager.cc:1950] T 30a602767b4a43eda51a53b5f4ebca4c P 29ff8298ac0d407386639b3834a51435: Deleting consensus metadata
I20251210 14:58:47.764932 15317 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: No bootstrap required, opened a new log
I20251210 14:58:47.765033 15317 ts_tablet_manager.cc:1403] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:47.765049 15318 tablet_bootstrap.cc:654] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.765570 15317 raft_consensus.cc:359] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.765704 15317 raft_consensus.cc:385] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.765743 15317 raft_consensus.cc:740] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.765877 15317 consensus_queue.cc:260] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.766201 15317 ts_tablet_manager.cc:1434] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:47.766554 15318 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: No bootstrap required, opened a new log
I20251210 14:58:47.766630 15318 ts_tablet_manager.cc:1403] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:47.766693 14785 catalog_manager.cc:4985] TS 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167): tablet 30a602767b4a43eda51a53b5f4ebca4c (table test-table [id=41c00058a2f34c5190ee4fefd049aaf6]) successfully deleted
I20251210 14:58:47.767154 15318 raft_consensus.cc:359] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.767248 15318 raft_consensus.cc:385] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.767280 15318 raft_consensus.cc:740] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.767426 15318 consensus_queue.cc:260] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.767541 15319 tablet_bootstrap.cc:492] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: No bootstrap required, opened a new log
I20251210 14:58:47.767606 15319 ts_tablet_manager.cc:1403] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251210 14:58:47.767656 15318 ts_tablet_manager.cc:1434] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:47.767704 14787 catalog_manager.cc:4985] TS 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841): tablet 30a602767b4a43eda51a53b5f4ebca4c (table test-table [id=41c00058a2f34c5190ee4fefd049aaf6]) successfully deleted
I20251210 14:58:47.768016 15319 raft_consensus.cc:359] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.768146 15319 raft_consensus.cc:385] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.768179 15319 raft_consensus.cc:740] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.768294 15319 consensus_queue.cc:260] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.768556 15319 ts_tablet_manager.cc:1434] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:47.776465 15342 ts_tablet_manager.cc:1929] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:47.776577 15342 log.cc:1199] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/30a602767b4a43eda51a53b5f4ebca4c
I20251210 14:58:47.776831 15342 ts_tablet_manager.cc:1950] T 30a602767b4a43eda51a53b5f4ebca4c P 21d05794bfb44748b737d4086d0f4d39: Deleting consensus metadata
I20251210 14:58:47.777822 14784 catalog_manager.cc:4985] TS 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725): tablet 30a602767b4a43eda51a53b5f4ebca4c (table test-table [id=41c00058a2f34c5190ee4fefd049aaf6]) successfully deleted
W20251210 14:58:47.841065 15290 tablet.cc:2378] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:58:47.847816 15324 raft_consensus.cc:493] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:47.847939 15324 raft_consensus.cc:515] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.849005 15324 leader_election.cc:290] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:47.853005 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.849234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:41841 (local address 127.6.115.66:44803)
1210 14:58:47.849535 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.849551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.849640 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.850465 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.850468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.850485 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.850773 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.850780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.851718 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.851721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.852633 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.852641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.852759 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.852782 (+ 23us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.852830 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.852887 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20251210 14:58:47.853849 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.849809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40585 (local address 127.6.115.67:40725)
1210 14:58:47.849938 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.849944 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.849959 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:58:47.850020 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.850023 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.850073 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.850151 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.850811 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.851553 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.852810 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.853343 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.853419 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.853658 (+ 239us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.853723 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:58:47.854167 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.849976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:44803 (local address 127.6.115.65:41841)
1210 14:58:47.850174 (+ 198us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.850178 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.850209 (+ 31us) server_negotiation.cc:408] Connection header received
1210 14:58:47.850257 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.850260 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.850305 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.850372 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:47.850899 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.851590 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.853472 (+ 1882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.853925 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.853958 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.854002 (+ 44us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.854059 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20251210 14:58:47.854241 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.849280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.6.115.66:40585)
1210 14:58:47.849709 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.849721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.849815 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.850285 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.850289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.850308 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:47.850635 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.850643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.851709 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.851713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.852633 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.852644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.853499 (+ 855us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.853517 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.854039 (+ 522us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.854102 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":202,"threads_started":1}
I20251210 14:58:47.854538 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aaa4e86b8784f159d6aa8d5100f9802" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435" is_pre_election: true
I20251210 14:58:47.854682 14959 raft_consensus.cc:2468] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 0.
I20251210 14:58:47.855006 15243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aaa4e86b8784f159d6aa8d5100f9802" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39" is_pre_election: true
I20251210 14:58:47.855091 15035 leader_election.cc:304] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ff8298ac0d407386639b3834a51435, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:47.855191 15243 raft_consensus.cc:2468] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 0.
I20251210 14:58:47.855310 15324 raft_consensus.cc:2804] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:47.855360 15324 raft_consensus.cc:493] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:47.855405 15324 raft_consensus.cc:3060] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.856345 15324 raft_consensus.cc:515] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.856693 15324 leader_election.cc:290] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 election: Requested vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:47.857129 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aaa4e86b8784f159d6aa8d5100f9802" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435"
I20251210 14:58:47.857175 15243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aaa4e86b8784f159d6aa8d5100f9802" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39"
I20251210 14:58:47.857249 14959 raft_consensus.cc:3060] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.857264 15243 raft_consensus.cc:3060] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.858227 14959 raft_consensus.cc:2468] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 1.
I20251210 14:58:47.858225 15243 raft_consensus.cc:2468] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 1.
I20251210 14:58:47.858528 15032 leader_election.cc:304] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:47.858705 15324 raft_consensus.cc:2804] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:47.858879 15324 raft_consensus.cc:697] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 LEADER]: Becoming Leader. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Running, Role: LEADER
I20251210 14:58:47.859127 15324 consensus_queue.cc:237] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.861388 14800 catalog_manager.cc:5654] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66). New cstate: current_term: 1 leader_uuid: "4bba6fe07ad648dcbcdefe5344066137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } health_report { overall_health: UNKNOWN } } }
W20251210 14:58:47.871716 15148 tablet.cc:2378] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251210 14:58:47.905411 15006 tablet.cc:2378] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Can't schedule compaction. Clean time has not been advanced past its initial value.
Dec 10 14:58:47 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378726, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.126@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:47.911435 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.903239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.6.115.126:43451)
1210 14:58:47.903360 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.903364 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.903544 (+ 180us) server_negotiation.cc:408] Connection header received
1210 14:58:47.903701 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.903704 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.903786 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.903855 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:47.904918 (+ 1063us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.905519 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.906231 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.906423 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.908533 (+ 2110us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:47.908547 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:47.908548 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:47.908567 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:47.910055 (+ 1488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.910429 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.910434 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.910435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.910468 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:47.910669 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:47.910672 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:47.910674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:47.910797 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:47.910876 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.911123 (+ 247us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.911297 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":49}
I20251210 14:58:47.911435 15354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.903104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:43451 (local address 127.0.0.1:58202)
1210 14:58:47.903434 (+ 330us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.903453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.903594 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.903920 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.903924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.904484 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:47.904771 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.904783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.905636 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.905639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.906118 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.906126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.906229 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.907002 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:47.907014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:47.908415 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:47.910154 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.910160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.910165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.910343 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.910543 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:47.910552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:47.910554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:47.910593 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:47.910893 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:47.910899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:47.910980 (+ 81us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.911070 (+ 90us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.911227 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20251210 14:58:47.915091 14800 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:58202:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251210 14:58:47.915736 14800 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:47.920696 14939 tablet_service.cc:1505] Processing CreateTablet for tablet 18aac379456046eb811b1854c9f6cc9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe088dbf08ef4bda827d49f2203d8a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:58:47.920763 15081 tablet_service.cc:1505] Processing CreateTablet for tablet 18aac379456046eb811b1854c9f6cc9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe088dbf08ef4bda827d49f2203d8a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:58:47.921100 14939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aac379456046eb811b1854c9f6cc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.921101 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aac379456046eb811b1854c9f6cc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.923946 15319 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: Bootstrap starting.
I20251210 14:58:47.924248 15318 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: Bootstrap starting.
I20251210 14:58:47.924440 15223 tablet_service.cc:1505] Processing CreateTablet for tablet 18aac379456046eb811b1854c9f6cc9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe088dbf08ef4bda827d49f2203d8a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251210 14:58:47.924767 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aac379456046eb811b1854c9f6cc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:47.925181 15319 tablet_bootstrap.cc:654] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.925231 15318 tablet_bootstrap.cc:654] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.927059 15319 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: No bootstrap required, opened a new log
I20251210 14:58:47.927142 15319 ts_tablet_manager.cc:1403] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251210 14:58:47.927433 15318 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: No bootstrap required, opened a new log
I20251210 14:58:47.927484 15317 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: Bootstrap starting.
I20251210 14:58:47.927531 15318 ts_tablet_manager.cc:1403] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:47.927593 15319 raft_consensus.cc:359] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.927690 15319 raft_consensus.cc:385] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.927719 15319 raft_consensus.cc:740] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.927968 15319 consensus_queue.cc:260] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.928056 15318 raft_consensus.cc:359] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.928165 15318 raft_consensus.cc:385] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.928200 15318 raft_consensus.cc:740] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.928186 15324 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.928301 15324 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.928319 15318 consensus_queue.cc:260] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.928382 15317 tablet_bootstrap.cc:654] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:47.928597 15318 ts_tablet_manager.cc:1434] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251210 14:58:47.928876 15325 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.928932 15319 ts_tablet_manager.cc:1434] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251210 14:58:47.928997 15325 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.930166 15317 tablet_bootstrap.cc:492] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: No bootstrap required, opened a new log
I20251210 14:58:47.930267 15317 ts_tablet_manager.cc:1403] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:47.930676 15317 raft_consensus.cc:359] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.930779 15317 raft_consensus.cc:385] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:47.930810 15317 raft_consensus.cc:740] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Initialized, Role: FOLLOWER
I20251210 14:58:47.930919 15317 consensus_queue.cc:260] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.931142 15335 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.931217 15317 ts_tablet_manager.cc:1434] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251210 14:58:47.931265 15335 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.963279 15335 raft_consensus.cc:493] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:47.963419 15335 raft_consensus.cc:515] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.963809 15335 leader_election.cc:290] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167)
I20251210 14:58:47.964262 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aac379456046eb811b1854c9f6cc9e" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435" is_pre_election: true
I20251210 14:58:47.964282 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aac379456046eb811b1854c9f6cc9e" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137" is_pre_election: true
I20251210 14:58:47.964416 14959 raft_consensus.cc:2468] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 0.
I20251210 14:58:47.964449 15101 raft_consensus.cc:2468] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 0.
I20251210 14:58:47.964708 15177 leader_election.cc:304] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 29ff8298ac0d407386639b3834a51435; no voters:
I20251210 14:58:47.964936 15335 raft_consensus.cc:2804] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:47.964986 15335 raft_consensus.cc:493] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:47.965057 15335 raft_consensus.cc:3060] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.966146 15335 raft_consensus.cc:515] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.966462 15335 leader_election.cc:290] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 election: Requested vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167)
I20251210 14:58:47.966843 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aac379456046eb811b1854c9f6cc9e" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435"
I20251210 14:58:47.966881 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aac379456046eb811b1854c9f6cc9e" candidate_uuid: "21d05794bfb44748b737d4086d0f4d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137"
I20251210 14:58:47.966941 14959 raft_consensus.cc:3060] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.966993 15101 raft_consensus.cc:3060] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:47.967815 15101 raft_consensus.cc:2468] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 1.
I20251210 14:58:47.967903 14959 raft_consensus.cc:2468] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d05794bfb44748b737d4086d0f4d39 in term 1.
I20251210 14:58:47.968137 15175 leader_election.cc:304] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:47.968329 15335 raft_consensus.cc:2804] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:47.968421 15335 raft_consensus.cc:697] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [term 1 LEADER]: Becoming Leader. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Running, Role: LEADER
I20251210 14:58:47.968571 15335 consensus_queue.cc:237] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:47.969169 15327 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: TxnStatusTablet state changed. Reason: New leader 21d05794bfb44748b737d4086d0f4d39. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.969254 15327 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:58:47.969532 15359 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251210 14:58:47.969687 15359 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251210 14:58:47.970266 14800 catalog_manager.cc:5654] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67). New cstate: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } health_report { overall_health: HEALTHY } } }
I20251210 14:58:47.980443 15354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.977684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.0.0.1:52372)
1210 14:58:47.977880 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:47.977892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:47.977972 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:47.978211 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:47.978213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:47.978219 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:58:47.978382 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.978386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.979299 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.979301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:47.979694 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:47.979700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.979812 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.979846 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:58:47.980235 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:58:47.980242 (+ 7us) client_negotiation.cc:769] Sending connection context
1210 14:58:47.980290 (+ 48us) client_negotiation.cc:240] Negotiation successful
1210 14:58:47.980347 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20251210 14:58:47.980489 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.977783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.6.115.67:40725)
1210 14:58:47.977898 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.977902 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.977919 (+ 17us) server_negotiation.cc:408] Connection header received
1210 14:58:47.978069 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.978073 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.978125 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.978203 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:47.978478 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.979185 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.979826 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.980014 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.980061 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:47.980140 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:47.980223 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.980324 (+ 101us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.980372 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:58:47.982643 15101 raft_consensus.cc:1275] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Refusing update from remote peer 21d05794bfb44748b737d4086d0f4d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:47.982643 14959 raft_consensus.cc:1275] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Refusing update from remote peer 21d05794bfb44748b737d4086d0f4d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:47.983148 15327 consensus_queue.cc:1048] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:47.984942 15327 consensus_queue.cc:1048] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:47.986388 15325 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: TxnStatusTablet state changed. Reason: New leader 21d05794bfb44748b737d4086d0f4d39. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.986474 15325 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.987249 15335 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.987331 15335 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:58:47.987685 15324 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: TxnStatusTablet state changed. Reason: New leader 21d05794bfb44748b737d4086d0f4d39. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.987773 15324 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.988675 15330 mvcc.cc:204] Tried to move back new op lower bound from 7230991269813448704 to 7230991269760618496. Current Snapshot: MvccSnapshot[applied={T|T < 7230991269813448704 or (T in {7230991269813448704})}]
I20251210 14:58:47.990124 15327 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.990201 15327 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 21d05794bfb44748b737d4086d0f4d39: This TxnStatusTablet replica's current role is: LEADER
I20251210 14:58:47.991099 15331 mvcc.cc:204] Tried to move back new op lower bound from 7230991269813448704 to 7230991269760618496. Current Snapshot: MvccSnapshot[applied={T|T < 7230991269813448704 or (T in {7230991269813448704})}]
I20251210 14:58:47.991707 15325 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.991806 15325 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 29ff8298ac0d407386639b3834a51435: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.993717 15324 tablet_replica.cc:442] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21d05794bfb44748b737d4086d0f4d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } } }
I20251210 14:58:47.993829 15324 tablet_replica.cc:445] T 18aac379456046eb811b1854c9f6cc9e P 4bba6fe07ad648dcbcdefe5344066137: This TxnStatusTablet replica's current role is: FOLLOWER
I20251210 14:58:47.995978 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:47.992066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.6.115.66:43167)
1210 14:58:47.992183 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:47.992187 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:47.992198 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:47.992278 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:47.992281 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:47.992337 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:47.992441 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:47.992998 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.993685 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:47.994777 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:47.994966 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:47.995033 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:47.995569 (+ 536us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:47.995678 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:47.995803 (+ 125us) server_negotiation.cc:300] Negotiation successful
1210 14:58:47.995861 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251210 14:58:48.006013 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.003007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:57549 (local address 127.6.115.67:40725)
1210 14:58:48.003121 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.003126 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.003367 (+ 241us) server_negotiation.cc:408] Connection header received
1210 14:58:48.003486 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.003488 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.003530 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.003590 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.004026 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.004452 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.005344 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.005570 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.005617 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.005696 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.005759 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.005865 (+ 106us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.005902 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251210 14:58:48.006072 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.002848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.6.115.66:57549)
1210 14:58:48.003275 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.003290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.003400 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.003616 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.003619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.003627 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:58:48.003911 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.003918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.004551 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.004554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.005237 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.005247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.005407 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.005449 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.005785 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.005790 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.005853 (+ 63us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.005924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
I20251210 14:58:48.012748 14959 raft_consensus.cc:1275] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Refusing update from remote peer 4bba6fe07ad648dcbcdefe5344066137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.012918 15243 raft_consensus.cc:1275] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Refusing update from remote peer 4bba6fe07ad648dcbcdefe5344066137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.013290 15324 consensus_queue.cc:1048] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.013656 15347 consensus_queue.cc:1048] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.035151 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.031983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:43167 (local address 127.6.115.67:36965)
1210 14:58:48.032220 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.032236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.032319 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.032542 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.032545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.032553 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:58:48.032774 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.032784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.033833 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.033839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.034361 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.034370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.034474 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.034512 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.034908 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.034913 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.034981 (+ 68us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.035045 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20251210 14:58:48.035179 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.032094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:36965 (local address 127.6.115.66:43167)
1210 14:58:48.032264 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.032271 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.032283 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:48.032414 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.032418 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.032460 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.032521 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.032895 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.033700 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.034482 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.034675 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.034723 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.034802 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.034881 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.035010 (+ 129us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.035057 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251210 14:58:48.062462 14797 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.062633 14797 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.065032 14797 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Sending DeleteTablet for 3 replicas of tablet 4aaa4e86b8784f159d6aa8d5100f9802
I20251210 14:58:48.065866 15081 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39796
I20251210 14:58:48.066057 15341 tablet_replica.cc:333] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: stopping tablet replica
I20251210 14:58:48.066167 15341 raft_consensus.cc:2243] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:48.066407 14939 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39600
I20251210 14:58:48.066445 15341 raft_consensus.cc:2272] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.066578 15340 tablet_replica.cc:333] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: stopping tablet replica
I20251210 14:58:48.066675 15340 raft_consensus.cc:2243] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.066771 15340 raft_consensus.cc:2272] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.067215 15341 ts_tablet_manager.cc:1916] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.069937 15341 ts_tablet_manager.cc:1929] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:58:48.070012 15341 log.cc:1199] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/4aaa4e86b8784f159d6aa8d5100f9802
I20251210 14:58:48.070255 15341 ts_tablet_manager.cc:1950] T 4aaa4e86b8784f159d6aa8d5100f9802 P 4bba6fe07ad648dcbcdefe5344066137: Deleting consensus metadata
I20251210 14:58:48.070921 15223 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aaa4e86b8784f159d6aa8d5100f9802 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:52340
I20251210 14:58:48.071074 14785 catalog_manager.cc:4985] TS 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167): tablet 4aaa4e86b8784f159d6aa8d5100f9802 (table test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]) successfully deleted
I20251210 14:58:48.071208 15342 tablet_replica.cc:333] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: stopping tablet replica
I20251210 14:58:48.071346 15342 raft_consensus.cc:2243] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.071511 15342 raft_consensus.cc:2272] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.071827 15340 ts_tablet_manager.cc:1916] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.074486 15340 ts_tablet_manager.cc:1929] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:58:48.074571 15340 log.cc:1199] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/4aaa4e86b8784f159d6aa8d5100f9802
I20251210 14:58:48.074873 15340 ts_tablet_manager.cc:1950] T 4aaa4e86b8784f159d6aa8d5100f9802 P 29ff8298ac0d407386639b3834a51435: Deleting consensus metadata
I20251210 14:58:48.075703 14787 catalog_manager.cc:4985] TS 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841): tablet 4aaa4e86b8784f159d6aa8d5100f9802 (table test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]) successfully deleted
I20251210 14:58:48.076583 15342 ts_tablet_manager.cc:1916] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.079169 15342 ts_tablet_manager.cc:1929] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:58:48.079252 15342 log.cc:1199] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/4aaa4e86b8784f159d6aa8d5100f9802
I20251210 14:58:48.079517 15342 ts_tablet_manager.cc:1950] T 4aaa4e86b8784f159d6aa8d5100f9802 P 21d05794bfb44748b737d4086d0f4d39: Deleting consensus metadata
I20251210 14:58:48.080215 14784 catalog_manager.cc:4985] TS 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725): tablet 4aaa4e86b8784f159d6aa8d5100f9802 (table test-table [id=28179c77c0b34cd5b506a5ec9d9ecc26]) successfully deleted
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
W20251210 14:58:48.099887 14849 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:48.092737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.6.115.126:43451)
1210 14:58:48.092861 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.092865 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.093051 (+ 186us) server_negotiation.cc:408] Connection header received
1210 14:58:48.093166 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.093170 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.093220 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.093307 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:48.094350 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.095178 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.096096 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.096283 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.099746 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58206: BlockingRecv error: recv got EOF from 127.0.0.1:58206 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM
I20251210 14:58:48.110657 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.101582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.6.115.126:43451)
1210 14:58:48.101720 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.101725 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.102114 (+ 389us) server_negotiation.cc:408] Connection header received
1210 14:58:48.102210 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.102213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.102263 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.102356 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:48.103166 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.103890 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.104692 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.105281 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.107336 (+ 2055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:48.107355 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:48.107358 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:48.107385 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:48.109080 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:48.109565 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:48.109568 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:48.109571 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:48.109618 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:48.109871 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:48.109874 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:48.109876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:48.110055 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:48.110177 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.110397 (+ 220us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.110503 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:58:48.113222 14797 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:48.113801 14797 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:48.119067 14939 tablet_service.cc:1505] Processing CreateTablet for tablet 82142bef91a6435ea0537539f3d3a155 (DEFAULT_TABLE table=test-table [id=239d207570dd4983a29b03fca65166ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.119379 14939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82142bef91a6435ea0537539f3d3a155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.122026 15081 tablet_service.cc:1505] Processing CreateTablet for tablet 82142bef91a6435ea0537539f3d3a155 (DEFAULT_TABLE table=test-table [id=239d207570dd4983a29b03fca65166ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.122048 15223 tablet_service.cc:1505] Processing CreateTablet for tablet 82142bef91a6435ea0537539f3d3a155 (DEFAULT_TABLE table=test-table [id=239d207570dd4983a29b03fca65166ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.122287 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82142bef91a6435ea0537539f3d3a155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.122329 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82142bef91a6435ea0537539f3d3a155. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.124560 15319 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Bootstrap starting.
I20251210 14:58:48.125494 15317 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Bootstrap starting.
I20251210 14:58:48.126325 15317 tablet_bootstrap.cc:654] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.127563 15319 tablet_bootstrap.cc:654] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.128480 15317 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: No bootstrap required, opened a new log
I20251210 14:58:48.128566 15317 ts_tablet_manager.cc:1403] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251210 14:58:48.129014 15317 raft_consensus.cc:359] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.129130 15317 raft_consensus.cc:385] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.129168 15317 raft_consensus.cc:740] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.129289 15317 consensus_queue.cc:260] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.129709 15317 ts_tablet_manager.cc:1434] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251210 14:58:48.130383 15319 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: No bootstrap required, opened a new log
I20251210 14:58:48.130483 15319 ts_tablet_manager.cc:1403] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20251210 14:58:48.131007 15319 raft_consensus.cc:359] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.131130 15319 raft_consensus.cc:385] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.131177 15319 raft_consensus.cc:740] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.131320 15319 consensus_queue.cc:260] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.131896 15318 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Bootstrap starting.
I20251210 14:58:48.132881 15318 tablet_bootstrap.cc:654] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.134284 15319 ts_tablet_manager.cc:1434] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251210 14:58:48.142270 15318 tablet_bootstrap.cc:492] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: No bootstrap required, opened a new log
I20251210 14:58:48.142377 15318 ts_tablet_manager.cc:1403] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251210 14:58:48.142861 15318 raft_consensus.cc:359] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.142987 15318 raft_consensus.cc:385] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.143018 15318 raft_consensus.cc:740] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.143144 15318 consensus_queue.cc:260] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.143399 15318 ts_tablet_manager.cc:1434] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:48.322610 15324 raft_consensus.cc:493] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:48.322757 15324 raft_consensus.cc:515] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.323298 15324 leader_election.cc:290] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:48.323735 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82142bef91a6435ea0537539f3d3a155" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435" is_pre_election: true
I20251210 14:58:48.323797 15243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82142bef91a6435ea0537539f3d3a155" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39" is_pre_election: true
I20251210 14:58:48.323871 14959 raft_consensus.cc:2468] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 0.
I20251210 14:58:48.323962 15243 raft_consensus.cc:2468] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 0.
I20251210 14:58:48.324179 15035 leader_election.cc:304] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ff8298ac0d407386639b3834a51435, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:48.324369 15324 raft_consensus.cc:2804] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:48.324432 15324 raft_consensus.cc:493] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:48.324474 15324 raft_consensus.cc:3060] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.325471 15324 raft_consensus.cc:515] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.325806 15324 leader_election.cc:290] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 election: Requested vote from peers 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:48.326165 14959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82142bef91a6435ea0537539f3d3a155" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ff8298ac0d407386639b3834a51435"
I20251210 14:58:48.326181 15243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82142bef91a6435ea0537539f3d3a155" candidate_uuid: "4bba6fe07ad648dcbcdefe5344066137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39"
I20251210 14:58:48.326325 14959 raft_consensus.cc:3060] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.326333 15243 raft_consensus.cc:3060] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.327387 14959 raft_consensus.cc:2468] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 1.
I20251210 14:58:48.327387 15243 raft_consensus.cc:2468] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bba6fe07ad648dcbcdefe5344066137 in term 1.
I20251210 14:58:48.327680 15032 leader_election.cc:304] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d05794bfb44748b737d4086d0f4d39, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:48.327878 15324 raft_consensus.cc:2804] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:48.327978 15324 raft_consensus.cc:697] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 LEADER]: Becoming Leader. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Running, Role: LEADER
I20251210 14:58:48.328125 15324 consensus_queue.cc:237] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.329748 14797 catalog_manager.cc:5654] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66). New cstate: current_term: 1 leader_uuid: "4bba6fe07ad648dcbcdefe5344066137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:48.360368 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.357718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.6.115.66:43167)
1210 14:58:48.357865 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.357870 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.357883 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:48.357924 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.357927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.357969 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.358031 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.358448 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.358983 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.359685 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.359837 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.359884 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.359967 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.360040 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.360203 (+ 163us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.360254 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251210 14:58:48.362316 15243 raft_consensus.cc:1275] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Refusing update from remote peer 4bba6fe07ad648dcbcdefe5344066137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.362344 14959 raft_consensus.cc:1275] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Refusing update from remote peer 4bba6fe07ad648dcbcdefe5344066137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.362748 15368 consensus_queue.cc:1048] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.362980 15324 consensus_queue.cc:1048] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.373832 14800 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.373955 14800 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.375782 14800 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Sending DeleteTablet for 3 replicas of tablet 82142bef91a6435ea0537539f3d3a155
I20251210 14:58:48.376293 14939 tablet_service.cc:1552] Processing DeleteTablet for tablet 82142bef91a6435ea0537539f3d3a155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39600
I20251210 14:58:48.376363 15081 tablet_service.cc:1552] Processing DeleteTablet for tablet 82142bef91a6435ea0537539f3d3a155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39796
I20251210 14:58:48.376366 15223 tablet_service.cc:1552] Processing DeleteTablet for tablet 82142bef91a6435ea0537539f3d3a155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:52340
I20251210 14:58:48.376475 15340 tablet_replica.cc:333] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: stopping tablet replica
I20251210 14:58:48.376511 15341 tablet_replica.cc:333] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: stopping tablet replica
I20251210 14:58:48.376536 15342 tablet_replica.cc:333] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: stopping tablet replica
I20251210 14:58:48.376580 15340 raft_consensus.cc:2243] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.376597 15341 raft_consensus.cc:2243] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:48.376658 15342 raft_consensus.cc:2243] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.376696 15340 raft_consensus.cc:2272] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.376775 15341 raft_consensus.cc:2272] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.376777 15342 raft_consensus.cc:2272] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.377292 15340 ts_tablet_manager.cc:1916] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.377394 15341 ts_tablet_manager.cc:1916] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.377571 15342 ts_tablet_manager.cc:1916] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.377976 14800 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:58:48.378494 14800 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:58:48.379416 15340 ts_tablet_manager.cc:1929] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:48.379416 15341 ts_tablet_manager.cc:1929] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:48.379487 15341 log.cc:1199] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/82142bef91a6435ea0537539f3d3a155
I20251210 14:58:48.379487 15340 log.cc:1199] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-0/wal/wals/82142bef91a6435ea0537539f3d3a155
I20251210 14:58:48.379739 15341 ts_tablet_manager.cc:1950] T 82142bef91a6435ea0537539f3d3a155 P 4bba6fe07ad648dcbcdefe5344066137: Deleting consensus metadata
I20251210 14:58:48.379737 15340 ts_tablet_manager.cc:1950] T 82142bef91a6435ea0537539f3d3a155 P 29ff8298ac0d407386639b3834a51435: Deleting consensus metadata
I20251210 14:58:48.379747 15342 ts_tablet_manager.cc:1929] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:58:48.379832 15342 log.cc:1199] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/82142bef91a6435ea0537539f3d3a155
I20251210 14:58:48.380065 15342 ts_tablet_manager.cc:1950] T 82142bef91a6435ea0537539f3d3a155 P 21d05794bfb44748b737d4086d0f4d39: Deleting consensus metadata
I20251210 14:58:48.380481 14785 catalog_manager.cc:4985] TS 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167): tablet 82142bef91a6435ea0537539f3d3a155 (table test-table [id=239d207570dd4983a29b03fca65166ef]) successfully deleted
I20251210 14:58:48.380659 14784 catalog_manager.cc:4985] TS 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725): tablet 82142bef91a6435ea0537539f3d3a155 (table test-table [id=239d207570dd4983a29b03fca65166ef]) successfully deleted
I20251210 14:58:48.380676 14787 catalog_manager.cc:4985] TS 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841): tablet 82142bef91a6435ea0537539f3d3a155 (table test-table [id=239d207570dd4983a29b03fca65166ef]) successfully deleted
I20251210 14:58:48.383839 14939 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 (DEFAULT_TABLE table=test-table [id=d41ff0b1da3e4aba867a197c928cae6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.383853 15081 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 (DEFAULT_TABLE table=test-table [id=d41ff0b1da3e4aba867a197c928cae6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.384087 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9dd286798d4dd19930f0c1eb3dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.384087 14939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9dd286798d4dd19930f0c1eb3dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.386022 15318 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: Bootstrap starting.
I20251210 14:58:48.386433 15223 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 (DEFAULT_TABLE table=test-table [id=d41ff0b1da3e4aba867a197c928cae6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:58:48.386673 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9dd286798d4dd19930f0c1eb3dc265. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:48.386929 15318 tablet_bootstrap.cc:654] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.387516 15319 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Bootstrap starting.
I20251210 14:58:48.388319 15319 tablet_bootstrap.cc:654] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.388406 15317 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Bootstrap starting.
I20251210 14:58:48.389072 15318 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: No bootstrap required, opened a new log
I20251210 14:58:48.389139 15318 ts_tablet_manager.cc:1403] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251210 14:58:48.389257 15317 tablet_bootstrap.cc:654] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:48.389567 15318 raft_consensus.cc:359] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.389658 15318 raft_consensus.cc:385] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.389684 15318 raft_consensus.cc:740] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.389799 15318 consensus_queue.cc:260] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.390054 15318 ts_tablet_manager.cc:1434] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:48.390345 15317 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: No bootstrap required, opened a new log
I20251210 14:58:48.390424 15317 ts_tablet_manager.cc:1403] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251210 14:58:48.390852 15317 raft_consensus.cc:359] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.390960 15317 raft_consensus.cc:385] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.390995 15317 raft_consensus.cc:740] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d05794bfb44748b737d4086d0f4d39, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.391124 15317 consensus_queue.cc:260] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.391391 15317 ts_tablet_manager.cc:1434] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251210 14:58:48.394455 15325 raft_consensus.cc:493] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:58:48.394536 15325 raft_consensus.cc:515] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.395404 15325 leader_election.cc:290] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:48.398170 15319 tablet_bootstrap.cc:492] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: No bootstrap required, opened a new log
I20251210 14:58:48.398267 15319 ts_tablet_manager.cc:1403] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20251210 14:58:48.398725 15319 raft_consensus.cc:359] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.398830 15319 raft_consensus.cc:385] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:48.398867 15319 raft_consensus.cc:740] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bba6fe07ad648dcbcdefe5344066137, State: Initialized, Role: FOLLOWER
I20251210 14:58:48.398986 15319 consensus_queue.cc:260] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.399245 15319 ts_tablet_manager.cc:1434] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251210 14:58:48.399533 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.395885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:53053 (local address 127.6.115.66:43167)
1210 14:58:48.396104 (+ 219us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.396109 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.396121 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:48.396162 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.396165 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.396213 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.396287 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:48.396812 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.397537 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.398665 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.399126 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.399165 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.399372 (+ 207us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.399439 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20251210 14:58:48.399838 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.395573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:43167 (local address 127.6.115.65:53053)
1210 14:58:48.395799 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.395816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.395909 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.396311 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.396314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.396331 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:48.396604 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.396613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.397646 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.397649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.398514 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.398524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.399225 (+ 701us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.399243 (+ 18us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.399643 (+ 400us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.399713 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20251210 14:58:48.400262 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9dd286798d4dd19930f0c1eb3dc265" candidate_uuid: "29ff8298ac0d407386639b3834a51435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137" is_pre_election: true
I20251210 14:58:48.400374 15101 raft_consensus.cc:2468] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29ff8298ac0d407386639b3834a51435 in term 0.
I20251210 14:58:48.400522 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.396432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:36761 (local address 127.6.115.67:40725)
1210 14:58:48.396541 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.396544 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.396555 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:48.396590 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.396592 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.396630 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.396697 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:58:48.397277 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.397959 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.399468 (+ 1509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.399900 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.399930 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.400363 (+ 433us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.400422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:58:48.400810 14891 leader_election.cc:304] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ff8298ac0d407386639b3834a51435, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:48.401058 15325 raft_consensus.cc:2804] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:58:48.401120 15325 raft_consensus.cc:493] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:58:48.401165 15325 raft_consensus.cc:3060] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.401429 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.395621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.6.115.65:36761)
1210 14:58:48.395886 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.395900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.396003 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.396835 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.396838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.396863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:58:48.397154 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.397161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.398126 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.398129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.399330 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.399340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.400221 (+ 881us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.400237 (+ 16us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.400793 (+ 556us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.400854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20251210 14:58:48.402087 15325 raft_consensus.cc:515] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.402387 15325 leader_election.cc:290] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [CANDIDATE]: Term 1 election: Requested vote from peers 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167), 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725)
I20251210 14:58:48.402719 15101 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9dd286798d4dd19930f0c1eb3dc265" candidate_uuid: "29ff8298ac0d407386639b3834a51435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bba6fe07ad648dcbcdefe5344066137"
I20251210 14:58:48.402835 15101 raft_consensus.cc:3060] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.402935 15242 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9dd286798d4dd19930f0c1eb3dc265" candidate_uuid: "29ff8298ac0d407386639b3834a51435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39"
I20251210 14:58:48.403021 15242 raft_consensus.cc:3060] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:48.403026 15243 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9dd286798d4dd19930f0c1eb3dc265" candidate_uuid: "29ff8298ac0d407386639b3834a51435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d05794bfb44748b737d4086d0f4d39" is_pre_election: true
I20251210 14:58:48.403720 15101 raft_consensus.cc:2468] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ff8298ac0d407386639b3834a51435 in term 1.
I20251210 14:58:48.403916 15242 raft_consensus.cc:2468] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ff8298ac0d407386639b3834a51435 in term 1.
I20251210 14:58:48.404026 14891 leader_election.cc:304] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ff8298ac0d407386639b3834a51435, 4bba6fe07ad648dcbcdefe5344066137; no voters:
I20251210 14:58:48.404193 15325 raft_consensus.cc:2804] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:48.404392 15325 raft_consensus.cc:697] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 1 LEADER]: Becoming Leader. State: Replica: 29ff8298ac0d407386639b3834a51435, State: Running, Role: LEADER
I20251210 14:58:48.404619 15325 consensus_queue.cc:237] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } }
I20251210 14:58:48.406829 14797 catalog_manager.cc:5654] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29ff8298ac0d407386639b3834a51435 (127.6.115.65). New cstate: current_term: 1 leader_uuid: "29ff8298ac0d407386639b3834a51435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ff8298ac0d407386639b3834a51435" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 41841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 } health_report { overall_health: UNKNOWN } } }
I20251210 14:58:48.425984 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.422933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.6.115.65:41841)
1210 14:58:48.423199 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.423202 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.423216 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:58:48.423280 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.423282 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.423330 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.423423 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.423776 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.424281 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.424972 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.425122 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.425172 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.425640 (+ 468us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.425745 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.425830 (+ 85us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.425872 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20251210 14:58:48.434947 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.432673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:54871 (local address 127.6.115.67:40725)
1210 14:58:48.432845 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.432848 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.432900 (+ 52us) server_negotiation.cc:408] Connection header received
1210 14:58:48.433005 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.433008 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.433050 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.433112 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.433461 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.433910 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.434466 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.434578 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.434610 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.434662 (+ 52us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.434715 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.434802 (+ 87us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.434856 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20251210 14:58:48.434970 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.432568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:40725 (local address 127.6.115.65:54871)
1210 14:58:48.432828 (+ 260us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.432840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.432925 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.433125 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.433127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.433133 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:58:48.433335 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.433339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.433998 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.434000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.434384 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.434390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.434474 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.434506 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.434735 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.434738 (+ 3us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.434788 (+ 50us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.434853 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20251210 14:58:48.440595 15101 raft_consensus.cc:1275] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Refusing update from remote peer 29ff8298ac0d407386639b3834a51435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.440595 15242 raft_consensus.cc:1275] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Refusing update from remote peer 29ff8298ac0d407386639b3834a51435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:58:48.441031 15325 consensus_queue.cc:1048] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bba6fe07ad648dcbcdefe5344066137" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.441200 15396 consensus_queue.cc:1048] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d05794bfb44748b737d4086d0f4d39" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:58:48.456399 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.453802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:41841 (local address 127.6.115.67:53409)
1210 14:58:48.453984 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.453999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:48.454068 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:48.454335 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:48.454338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:48.454348 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1210 14:58:48.454514 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.454520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.455332 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.455335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:48.455726 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:48.455733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.455819 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.455841 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.456198 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.456202 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:58:48.456255 (+ 53us) client_negotiation.cc:240] Negotiation successful
1210 14:58:48.456308 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20251210 14:58:48.456404 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.453949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:53409 (local address 127.6.115.65:41841)
1210 14:58:48.454054 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:48.454057 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:48.454068 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:48.454202 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:48.454205 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:48.454243 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:48.454297 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:58:48.454630 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.455237 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:48.455817 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:48.456021 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:48.456063 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:58:48.456121 (+ 58us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:58:48.456180 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:48.456277 (+ 97us) server_negotiation.cc:300] Negotiation successful
1210 14:58:48.456312 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:58:48.476169 14800 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.476295 14800 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:58:48.478153 14800 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ca2f14f7a9144026af8265ac2ee887d6: Sending DeleteTablet for 3 replicas of tablet 2f9dd286798d4dd19930f0c1eb3dc265
I20251210 14:58:48.478595 14939 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39600
I20251210 14:58:48.478657 15081 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39796
I20251210 14:58:48.478753 15340 tablet_replica.cc:333] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: stopping tablet replica
I20251210 14:58:48.478837 15341 tablet_replica.cc:333] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: stopping tablet replica
I20251210 14:58:48.478849 15340 raft_consensus.cc:2243] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:58:48.478933 15341 raft_consensus.cc:2243] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.479038 15341 raft_consensus.cc:2272] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.479086 15340 raft_consensus.cc:2272] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.479441 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 14874
I20251210 14:58:48.479801 15340 ts_tablet_manager.cc:1916] T 2f9dd286798d4dd19930f0c1eb3dc265 P 29ff8298ac0d407386639b3834a51435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.479897 15341 ts_tablet_manager.cc:1916] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.482374 15223 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9dd286798d4dd19930f0c1eb3dc265 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:58:48 UTC) from {username='oryx', principal='oryx/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:52340
I20251210 14:58:48.482559 15342 tablet_replica.cc:333] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: stopping tablet replica
I20251210 14:58:48.482728 15342 raft_consensus.cc:2243] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:58:48.482853 15342 raft_consensus.cc:2272] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:58:48.483382 15342 ts_tablet_manager.cc:1916] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:58:48.485013 15342 ts_tablet_manager.cc:1929] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:58:48.485087 15342 log.cc:1199] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/2f9dd286798d4dd19930f0c1eb3dc265
I20251210 14:58:48.485313 15342 ts_tablet_manager.cc:1950] T 2f9dd286798d4dd19930f0c1eb3dc265 P 21d05794bfb44748b737d4086d0f4d39: Deleting consensus metadata
I20251210 14:58:48.485922 14784 catalog_manager.cc:4985] TS 21d05794bfb44748b737d4086d0f4d39 (127.6.115.67:40725): tablet 2f9dd286798d4dd19930f0c1eb3dc265 (table test-table [id=d41ff0b1da3e4aba867a197c928cae6a]) successfully deleted
I20251210 14:58:48.487820 15341 ts_tablet_manager.cc:1929] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251210 14:58:48.487881 15341 log.cc:1199] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/2f9dd286798d4dd19930f0c1eb3dc265
I20251210 14:58:48.488092 15341 ts_tablet_manager.cc:1950] T 2f9dd286798d4dd19930f0c1eb3dc265 P 4bba6fe07ad648dcbcdefe5344066137: Deleting consensus metadata
I20251210 14:58:48.488662 14785 catalog_manager.cc:4985] TS 4bba6fe07ad648dcbcdefe5344066137 (127.6.115.66:43167): tablet 2f9dd286798d4dd19930f0c1eb3dc265 (table test-table [id=d41ff0b1da3e4aba867a197c928cae6a]) successfully deleted
W20251210 14:58:48.488957 14787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:41841 (error 108)
I20251210 14:58:48.489020 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15016
I20251210 14:58:48.490262 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:48.489913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:41841 (local address 127.0.0.1:39618)
1210 14:58:48.490126 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:48.490179 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20251210 14:58:48.490398 14787 catalog_manager.cc:4712] TS 29ff8298ac0d407386639b3834a51435 (127.6.115.65:41841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f9dd286798d4dd19930f0c1eb3dc265: Network error: Client connection negotiation failed: client connection to 127.6.115.65:41841: connect: Connection refused (error 111)
I20251210 14:58:48.496901 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15158
I20251210 14:58:48.503422 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 14768
2025-12-10T14:58:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3927 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[15411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[15411](info): set up 2 sockets
Dec 10 14:58:48 dist-test-slave-nb51 krb5kdc[15411](info): commencing operation
krb5kdc: starting...
W20251210 14:58:50.494024 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.967s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:50 dist-test-slave-nb51 krb5kdc[15411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:50.622936 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39665
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:39665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:50.722404 15427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:50.722586 15427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:50.722638 15427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:50.725777 15427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:50.725823 15427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:50.725845 15427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:50.725869 15427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:50.725907 15427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:50.729902 15427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:39665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:50.730823 15427 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:50.731601 15427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:50.736739 15433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:50.736809 15432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:50.736775 15435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:50.736933 15427 server_base.cc:1047] running on GCE node
I20251210 14:58:50.737537 15427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:50.738298 15427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:50.739431 15427 hybrid_clock.cc:648] HybridClock initialized: now 1765378730739411 us; error 34 us; skew 500 ppm
W20251210 14:58:50.742192 15427 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-12-10T14:58:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2244 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:58:50 dist-test-slave-nb51 krb5kdc[15440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:58:50 dist-test-slave-nb51 krb5kdc[15440](info): set up 2 sockets
Dec 10 14:58:50 dist-test-slave-nb51 krb5kdc[15440](info): commencing operation
krb5kdc: starting...
W20251210 14:58:52.714066 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:58:52 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:58:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:52.840340 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40807
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:52.938429 15456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:52.938616 15456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:52.938668 15456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:52.941789 15456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:52.941840 15456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:52.941862 15456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:52.941880 15456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:52.941897 15456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:52.945873 15456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:52.946779 15456 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:52.947582 15456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:52.952804 15464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:52.952843 15462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:52.952865 15461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:52.952953 15456 server_base.cc:1047] running on GCE node
I20251210 14:58:52.953564 15456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:52.954377 15456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:52.955521 15456 hybrid_clock.cc:648] HybridClock initialized: now 1765378732955510 us; error 31 us; skew 500 ppm
Dec 10 14:58:52 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378732, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:52.957968 15456 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:52.958957 15456 webserver.cc:492] Webserver started at http://127.6.115.126:37557/ using document root <none> and password file <none>
I20251210 14:58:52.959441 15456 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:52.959487 15456 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:52.959661 15456 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:52.961263 15456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "eec56660b8514ff8b35177a031c12d4a"
format_stamp: "Formatted at 2025-12-10 14:58:52 on dist-test-slave-nb51"
server_key: "816fcac33db5b86ee7e6be5721ecc3a6"
server_key_iv: "703477fa127c070b8926fea0bbd4ed96"
server_key_version: "encryptionkey@0"
I20251210 14:58:52.961773 15456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "eec56660b8514ff8b35177a031c12d4a"
format_stamp: "Formatted at 2025-12-10 14:58:52 on dist-test-slave-nb51"
server_key: "816fcac33db5b86ee7e6be5721ecc3a6"
server_key_iv: "703477fa127c070b8926fea0bbd4ed96"
server_key_version: "encryptionkey@0"
I20251210 14:58:52.964893 15456 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251210 14:58:52.966922 15471 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:52.967914 15456 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:58:52.968011 15456 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "eec56660b8514ff8b35177a031c12d4a"
format_stamp: "Formatted at 2025-12-10 14:58:52 on dist-test-slave-nb51"
server_key: "816fcac33db5b86ee7e6be5721ecc3a6"
server_key_iv: "703477fa127c070b8926fea0bbd4ed96"
server_key_version: "encryptionkey@0"
I20251210 14:58:52.968084 15456 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:52.999151 15456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:53.002092 15456 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:53.002293 15456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:53.010087 15456 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:40807
I20251210 14:58:53.010113 15523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:40807 every 8 connection(s)
I20251210 14:58:53.011030 15456 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:53.013764 15524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:58:53.015015 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15456
I20251210 14:58:53.015116 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:58:53.015347 6605 external_mini_cluster.cc:1468] Setting key ab45e0e9179f9244cdcc947d0bc6e98c
I20251210 14:58:53.020098 15524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Bootstrap starting.
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:53.023066 15524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Neither blocks nor log segments found. Creating new log.
I20251210 14:58:53.023841 15524 log.cc:826] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:53.025981 15524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: No bootstrap required, opened a new log
I20251210 14:58:53.028182 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.016525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.6.115.126:40807)
1210 14:58:53.016990 (+ 465us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:53.017002 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:53.017044 (+ 42us) server_negotiation.cc:408] Connection header received
1210 14:58:53.017746 (+ 702us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:53.017784 (+ 38us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:53.018115 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:53.018457 (+ 342us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:53.019151 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.020272 (+ 1121us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.020922 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.021241 (+ 319us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.023570 (+ 2329us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:53.023589 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:53.023601 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:53.023643 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:53.025878 (+ 2235us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.026318 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.026322 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.026326 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.026392 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.026620 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.026623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.026625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.026925 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:53.027126 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:53.027431 (+ 305us) server_negotiation.cc:300] Negotiation successful
1210 14:58:53.027643 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20251210 14:58:53.028502 15524 raft_consensus.cc:359] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.028678 15524 raft_consensus.cc:385] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:58:53.028743 15524 raft_consensus.cc:740] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec56660b8514ff8b35177a031c12d4a, State: Initialized, Role: FOLLOWER
I20251210 14:58:53.029172 15524 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.029333 15524 raft_consensus.cc:399] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:53.029435 15524 raft_consensus.cc:493] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:53.029528 15524 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:58:53.030694 15524 raft_consensus.cc:515] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.030967 15524 leader_election.cc:304] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec56660b8514ff8b35177a031c12d4a; no voters:
I20251210 14:58:53.031208 15524 leader_election.cc:290] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:58:53.031307 15529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:58:53.031544 15529 raft_consensus.cc:697] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 LEADER]: Becoming Leader. State: Replica: eec56660b8514ff8b35177a031c12d4a, State: Running, Role: LEADER
I20251210 14:58:53.031858 15529 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.032299 15524 sys_catalog.cc:565] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:53.032903 15529 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec56660b8514ff8b35177a031c12d4a. Latest consensus state: current_term: 1 leader_uuid: "eec56660b8514ff8b35177a031c12d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } } }
I20251210 14:58:53.033109 15529 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:53.033239 15530 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eec56660b8514ff8b35177a031c12d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } } }
I20251210 14:58:53.033387 15530 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: This master's current role is: LEADER
W20251210 14:58:53.036666 15545 catalog_manager.cc:1568] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251210 14:58:53.036726 15545 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251210 14:58:53.036859 15537 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:53.037815 15537 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:53.042650 15537 catalog_manager.cc:1357] Generated new cluster ID: 885166e98685426685832a95f63c0aec
I20251210 14:58:53.042704 15537 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:53.050233 15537 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:58:53.051051 15537 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:53.059271 15537 catalog_manager.cc:6027] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Generated new TSK 0
I20251210 14:58:53.059814 15537 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:53.108554 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40807
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:53.208701 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:53.208880 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:53.208933 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:53.212023 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:53.212076 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:53.212167 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:53.216050 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:53.217003 15552 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:53.217784 15552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:53.223776 15558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.223786 15557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.223786 15560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:53.223995 15552 server_base.cc:1047] running on GCE node
I20251210 14:58:53.224424 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:53.224920 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:53.226071 15552 hybrid_clock.cc:648] HybridClock initialized: now 1765378733226060 us; error 24 us; skew 500 ppm
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:53.228507 15552 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:58:53.229547 15552 webserver.cc:492] Webserver started at http://127.6.115.65:42943/ using document root <none> and password file <none>
I20251210 14:58:53.230039 15552 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:53.230087 15552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:53.230252 15552 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:53.231694 15552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "ae0b16dce3a24d96844c358d3663a38d"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "a7e28b3294b132ad43fe72ae1dd965ea"
server_key_iv: "c1fa2e75c7d0d5d1485be0c0a756ae6c"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.232151 15552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "ae0b16dce3a24d96844c358d3663a38d"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "a7e28b3294b132ad43fe72ae1dd965ea"
server_key_iv: "c1fa2e75c7d0d5d1485be0c0a756ae6c"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.235378 15552 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251210 14:58:53.237381 15567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.238332 15552 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251210 14:58:53.238430 15552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "ae0b16dce3a24d96844c358d3663a38d"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "a7e28b3294b132ad43fe72ae1dd965ea"
server_key_iv: "c1fa2e75c7d0d5d1485be0c0a756ae6c"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.238517 15552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:53.252609 15552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:53.255298 15552 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:53.255465 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:53.255971 15552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:53.256729 15552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:53.256775 15552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.256829 15552 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:53.256862 15552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.265799 15552 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:43509
I20251210 14:58:53.265823 15680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:43509 every 8 connection(s)
I20251210 14:58:53.266625 15552 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:53.273478 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15552
I20251210 14:58:53.273628 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:58:53.273993 6605 external_mini_cluster.cc:1468] Setting key 8dc8a118be9b188769d4588437f34fc0
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:53.279103 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.268314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:59727 (local address 127.6.115.126:40807)
1210 14:58:53.268474 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:53.268479 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:53.269066 (+ 587us) server_negotiation.cc:408] Connection header received
1210 14:58:53.269929 (+ 863us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:53.269933 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:53.269998 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:53.270082 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:53.271632 (+ 1550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.272197 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.272886 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.273103 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.275496 (+ 2393us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:53.275511 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:53.275513 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:53.275535 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:53.276898 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.277592 (+ 694us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.277595 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.277597 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.277654 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.278061 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.278064 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.278066 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.278211 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:53.278302 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:53.278869 (+ 567us) server_negotiation.cc:300] Negotiation successful
1210 14:58:53.278992 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251210 14:58:53.279925 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.268505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:59727)
1210 14:58:53.268923 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:53.268955 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:53.269730 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:53.270240 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:53.270260 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:53.270599 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:53.271465 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.271479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.272336 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.272339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:53.272752 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.272760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.272975 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.273674 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:53.273712 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:53.275349 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:53.277015 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.277020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.277029 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.277469 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.277787 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.277793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.277797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.277913 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.278321 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:53.278331 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:53.278594 (+ 263us) client_negotiation.cc:769] Sending connection context
1210 14:58:53.278800 (+ 206us) client_negotiation.cc:240] Negotiation successful
1210 14:58:53.279068 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20251210 14:58:53.281121 15681 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40807
I20251210 14:58:53.281344 15681 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:53.281809 15681 heartbeater.cc:507] Master 127.6.115.126:40807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:58:53.283154 15486 ts_manager.cc:194] Registered new tserver with Master: ae0b16dce3a24d96844c358d3663a38d (127.6.115.65:43509)
I20251210 14:58:53.284267 15486 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:59727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:53.320523 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40807
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:53.419620 15688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:53.419796 15688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:53.419885 15688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:53.423003 15688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:53.423061 15688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:53.423175 15688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:53.427011 15688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:53.427894 15688 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:53.428597 15688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:53.434443 15693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.434439 15694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.434471 15696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:53.434818 15688 server_base.cc:1047] running on GCE node
I20251210 14:58:53.435142 15688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:53.435611 15688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:53.436759 15688 hybrid_clock.cc:648] HybridClock initialized: now 1765378733436752 us; error 36 us; skew 500 ppm
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:53.439160 15688 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:58:53.440124 15688 webserver.cc:492] Webserver started at http://127.6.115.66:37209/ using document root <none> and password file <none>
I20251210 14:58:53.440623 15688 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:53.440676 15688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:53.440841 15688 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:53.442657 15688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "92a4920d66284c0aa8237d2b9cc38bf3"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "70fdfa72d9414486562509c5fb5bd863"
server_key_iv: "ba2fdc6bc001b785bc6066337f31f857"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.443142 15688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "92a4920d66284c0aa8237d2b9cc38bf3"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "70fdfa72d9414486562509c5fb5bd863"
server_key_iv: "ba2fdc6bc001b785bc6066337f31f857"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.446648 15688 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:58:53.448649 15703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.449618 15688 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:53.449714 15688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "92a4920d66284c0aa8237d2b9cc38bf3"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "70fdfa72d9414486562509c5fb5bd863"
server_key_iv: "ba2fdc6bc001b785bc6066337f31f857"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.449793 15688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:53.462291 15688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:53.465584 15688 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:53.465739 15688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:53.466249 15688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:53.467095 15688 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:53.467137 15688 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.467177 15688 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:53.467196 15688 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.476724 15688 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:38037
I20251210 14:58:53.476763 15816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:38037 every 8 connection(s)
I20251210 14:58:53.477582 15688 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:53.479784 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15688
I20251210 14:58:53.479907 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:58:53.480159 6605 external_mini_cluster.cc:1468] Setting key 5ad7d058f36b6eac7c0f23efd171f249
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
I20251210 14:58:53.489724 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.479149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:59873 (local address 127.6.115.126:40807)
1210 14:58:53.479301 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:53.479304 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:53.479890 (+ 586us) server_negotiation.cc:408] Connection header received
1210 14:58:53.480674 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:53.480677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:53.480737 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:53.480814 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:53.482412 (+ 1598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.483166 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.483845 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.484072 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.486274 (+ 2202us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:53.486292 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:53.486295 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:53.486319 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:53.487780 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.488359 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.488361 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.488363 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.488404 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.488716 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.488719 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.488721 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.488856 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:53.488934 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:53.489402 (+ 468us) server_negotiation.cc:300] Negotiation successful
1210 14:58:53.489520 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251210 14:58:53.490145 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.479382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:59873)
1210 14:58:53.479765 (+ 383us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:53.479797 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:53.480492 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:53.480966 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:53.480973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:53.481334 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:53.482256 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.482269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.483311 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.483315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:53.483725 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.483740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.483907 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.484474 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:53.484499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:53.486092 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:53.487924 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.487929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.487939 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.488260 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.488493 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.488496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.488499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.488607 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.488963 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:53.488969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:53.489153 (+ 184us) client_negotiation.cc:769] Sending connection context
1210 14:58:53.489310 (+ 157us) client_negotiation.cc:240] Negotiation successful
1210 14:58:53.489534 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
I20251210 14:58:53.491158 15817 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40807
I20251210 14:58:53.491386 15817 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:53.491832 15817 heartbeater.cc:507] Master 127.6.115.126:40807 requested a full tablet report, sending...
I20251210 14:58:53.492795 15486 ts_manager.cc:194] Registered new tserver with Master: 92a4920d66284c0aa8237d2b9cc38bf3 (127.6.115.66:38037)
I20251210 14:58:53.493324 15486 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:59873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:58:53.524592 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40807
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:53.621793 15824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:53.621970 15824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:53.622056 15824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:53.625010 15824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:53.625064 15824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:53.625178 15824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:53.628959 15824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:53.629879 15824 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:53.630632 15824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:53.636502 15829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.636530 15832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:53.636615 15824 server_base.cc:1047] running on GCE node
W20251210 14:58:53.636502 15830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:53.637053 15824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:53.637626 15824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:53.638787 15824 hybrid_clock.cc:648] HybridClock initialized: now 1765378733638773 us; error 29 us; skew 500 ppm
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:53.641440 15824 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:58:53.642431 15824 webserver.cc:492] Webserver started at http://127.6.115.67:39187/ using document root <none> and password file <none>
I20251210 14:58:53.642937 15824 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:53.642987 15824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:53.643158 15824 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:58:53.644740 15824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "133a4b8e05ae4652a1477951cdba6b00"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "7b56f9eea2c866435279c69fd3408d56"
server_key_iv: "b5584213576e5b16441aa049dcf4e07a"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.645177 15824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "133a4b8e05ae4652a1477951cdba6b00"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "7b56f9eea2c866435279c69fd3408d56"
server_key_iv: "b5584213576e5b16441aa049dcf4e07a"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.648283 15824 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251210 14:58:53.650327 15839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.651300 15824 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:53.651398 15824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "133a4b8e05ae4652a1477951cdba6b00"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "7b56f9eea2c866435279c69fd3408d56"
server_key_iv: "b5584213576e5b16441aa049dcf4e07a"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.651500 15824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:53.671151 15824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:53.673791 15824 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:53.673956 15824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:53.674433 15824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:53.675235 15824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:53.675282 15824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.675339 15824 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:53.675367 15824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.683904 15824 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:41985
I20251210 14:58:53.683962 15952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:41985 every 8 connection(s)
I20251210 14:58:53.684794 15824 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:58:53.689256 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15824
I20251210 14:58:53.689407 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:58:53.689684 6605 external_mini_cluster.cc:1468] Setting key 517cd3c488e24c697853ecb5f96aa77c
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:58:53.696635 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.686377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:59293 (local address 127.6.115.126:40807)
1210 14:58:53.686523 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:53.686526 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:53.687109 (+ 583us) server_negotiation.cc:408] Connection header received
1210 14:58:53.687943 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:53.687947 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:53.688023 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:53.688110 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:53.689929 (+ 1819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.690562 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.691513 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.691697 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.693763 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:53.693777 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:53.693779 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:53.693801 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:53.695138 (+ 1337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.695556 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.695559 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.695560 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.695594 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:53.695814 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:53.695816 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:53.695817 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:53.695936 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:53.696003 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:53.696413 (+ 410us) server_negotiation.cc:300] Negotiation successful
1210 14:58:53.696530 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251210 14:58:53.697247 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:53.686614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:59293)
1210 14:58:53.686984 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:53.687015 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:53.687743 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:53.688286 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:53.688294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:53.688644 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:53.689768 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.689786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.690781 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:53.690785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:53.691397 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:53.691406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:53.691602 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:53.692168 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:53.692212 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:53.693611 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:58:53.695255 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.695259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.695266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.695477 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.695664 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:58:53.695667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:58:53.695668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:58:53.695746 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:58:53.696006 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:58:53.696011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:58:53.696231 (+ 220us) client_negotiation.cc:769] Sending connection context
1210 14:58:53.696383 (+ 152us) client_negotiation.cc:240] Negotiation successful
1210 14:58:53.696597 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20251210 14:58:53.698180 15953 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40807
I20251210 14:58:53.698406 15953 heartbeater.cc:461] Registering TS with master...
I20251210 14:58:53.698894 15953 heartbeater.cc:507] Master 127.6.115.126:40807 requested a full tablet report, sending...
I20251210 14:58:53.699790 15486 ts_manager.cc:194] Registered new tserver with Master: 133a4b8e05ae4652a1477951cdba6b00 (127.6.115.67:41985)
I20251210 14:58:53.700276 15486 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:59293
I20251210 14:58:53.703245 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.65.keytab.
Entry for principal oryx/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.65.keytab.
WARNING: no policy specified for oryx/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.66.keytab.
Entry for principal oryx/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.66.keytab.
WARNING: no policy specified for oryx/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.67.keytab.
Entry for principal oryx/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.67.keytab.
I20251210 14:58:53.767903 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15552
I20251210 14:58:53.773782 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15688
I20251210 14:58:53.779145 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15824
I20251210 14:58:53.784539 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15456
I20251210 14:58:53.790498 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40807
--webserver_interface=127.6.115.126
--webserver_port=37557
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:58:53.890025 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:53.890224 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:53.890303 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:53.893386 15962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:58:53.893441 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:53.893481 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:53.893520 15962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:58:53.893554 15962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:58:53.897466 15962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=37557
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:53.898352 15962 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:53.899082 15962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251210 14:58:53.904016 15962 server_base.cc:1047] running on GCE node
W20251210 14:58:53.903962 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.903964 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:53.903962 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:53.904672 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:53.905572 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:53.906729 15962 hybrid_clock.cc:648] HybridClock initialized: now 1765378733906721 us; error 25 us; skew 500 ppm
Dec 10 14:58:53 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378733, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:53.909077 15962 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:58:53.910100 15962 webserver.cc:492] Webserver started at http://127.6.115.126:37557/ using document root <none> and password file <none>
I20251210 14:58:53.910562 15962 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:53.910616 15962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:53.913697 15962 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:53.915453 15977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:53.916324 15962 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:53.916409 15962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "eec56660b8514ff8b35177a031c12d4a"
format_stamp: "Formatted at 2025-12-10 14:58:52 on dist-test-slave-nb51"
server_key: "816fcac33db5b86ee7e6be5721ecc3a6"
server_key_iv: "703477fa127c070b8926fea0bbd4ed96"
server_key_version: "encryptionkey@0"
I20251210 14:58:53.916762 15962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:53.932022 15962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:53.934716 15962 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:53.934840 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:53.941640 15962 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:40807
I20251210 14:58:53.941668 16029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:40807 every 8 connection(s)
I20251210 14:58:53.942595 15962 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:58:53.945034 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 15962
I20251210 14:58:53.945547 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:43509
--local_ip_for_outbound_sockets=127.6.115.65
--tserver_master_addrs=127.6.115.126:40807
--webserver_port=42943
--webserver_interface=127.6.115.65
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.115.65
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251210 14:58:53.947491 16030 sys_catalog.cc:263] Verifying existing consensus state
I20251210 14:58:53.948989 16030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Bootstrap starting.
I20251210 14:58:53.964219 16030 log.cc:826] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Log is configured to *not* fsync() on all Append() calls
I20251210 14:58:53.970304 16030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251210 14:58:53.970643 16030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Bootstrap complete.
I20251210 14:58:53.974040 16030 raft_consensus.cc:359] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.974321 16030 raft_consensus.cc:740] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec56660b8514ff8b35177a031c12d4a, State: Initialized, Role: FOLLOWER
I20251210 14:58:53.974782 16030 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.974901 16030 raft_consensus.cc:399] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:58:53.974963 16030 raft_consensus.cc:493] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:58:53.975046 16030 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 1 FOLLOWER]: Advancing to term 2
I20251210 14:58:53.976430 16030 raft_consensus.cc:515] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.976713 16030 leader_election.cc:304] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec56660b8514ff8b35177a031c12d4a; no voters:
I20251210 14:58:53.976950 16030 leader_election.cc:290] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [CANDIDATE]: Term 2 election: Requested vote from peers
I20251210 14:58:53.977049 16034 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 2 FOLLOWER]: Leader election won for term 2
I20251210 14:58:53.977216 16034 raft_consensus.cc:697] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [term 2 LEADER]: Becoming Leader. State: Replica: eec56660b8514ff8b35177a031c12d4a, State: Running, Role: LEADER
I20251210 14:58:53.977512 16034 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } }
I20251210 14:58:53.978019 16030 sys_catalog.cc:565] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:58:53.978345 16036 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec56660b8514ff8b35177a031c12d4a. Latest consensus state: current_term: 2 leader_uuid: "eec56660b8514ff8b35177a031c12d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } } }
I20251210 14:58:53.978449 16036 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:53.978660 16035 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eec56660b8514ff8b35177a031c12d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec56660b8514ff8b35177a031c12d4a" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40807 } } }
I20251210 14:58:53.978755 16035 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a [sys.catalog]: This master's current role is: LEADER
I20251210 14:58:53.979193 16038 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:58:53.982125 16038 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:58:53.982780 16038 catalog_manager.cc:1269] Loaded cluster ID: 885166e98685426685832a95f63c0aec
I20251210 14:58:53.982844 16038 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:58:53.985221 16038 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:58:53.985908 16038 catalog_manager.cc:6038] T 00000000000000000000000000000000 P eec56660b8514ff8b35177a031c12d4a: Loaded TSK: 0
I20251210 14:58:53.987066 16038 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251210 14:58:54.048857 16032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:54.049062 16032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:54.049145 16032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:54.052367 16032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:54.052434 16032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:54.052582 16032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:58:54.056630 16032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.65.keytab
--principal=oryx/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:43509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=42943
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:54.057560 16032 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:54.058344 16032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:54.064033 16057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:54.064172 16059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:54.064036 16056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:54.064806 16032 server_base.cc:1047] running on GCE node
I20251210 14:58:54.065219 16032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:54.065838 16032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:54.067016 16032 hybrid_clock.cc:648] HybridClock initialized: now 1765378734067005 us; error 32 us; skew 500 ppm
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378734, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:54.069540 16032 init.cc:377] Logged in from keytab as oryx/127.6.115.65@KRBTEST.COM (short username oryx)
I20251210 14:58:54.070580 16032 webserver.cc:492] Webserver started at http://127.6.115.65:42943/ using document root <none> and password file <none>
I20251210 14:58:54.071095 16032 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:54.071162 16032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:54.074438 16032 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:54.076279 16066 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.077246 16032 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:54.077340 16032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "ae0b16dce3a24d96844c358d3663a38d"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "a7e28b3294b132ad43fe72ae1dd965ea"
server_key_iv: "c1fa2e75c7d0d5d1485be0c0a756ae6c"
server_key_version: "encryptionkey@0"
I20251210 14:58:54.077770 16032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:54.097452 16032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:54.100190 16032 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:54.100339 16032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:54.100850 16032 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:54.101704 16032 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:54.101751 16032 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.101788 16032 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:54.101805 16032 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.112340 16032 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:43509
I20251210 14:58:54.112455 16179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:43509 every 8 connection(s)
I20251210 14:58:54.113301 16032 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:58:54.121119 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16032
I20251210 14:58:54.121656 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:38037
--local_ip_for_outbound_sockets=127.6.115.66
--tserver_master_addrs=127.6.115.126:40807
--webserver_port=37209
--webserver_interface=127.6.115.66
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.115.66
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.126729 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.115416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:57677)
1210 14:58:54.115954 (+ 538us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.115991 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.116719 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.117933 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.117940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.118278 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.119138 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.119150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.120737 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.120740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.121308 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.121317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.121578 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.122251 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.122285 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.126033 (+ 3748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":117,"threads_started":1}
W20251210 14:58:54.126730 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.115406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:57677 (local address 127.6.115.126:40807)
1210 14:58:54.115851 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.115863 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.116112 (+ 249us) server_negotiation.cc:408] Connection header received
1210 14:58:54.117109 (+ 997us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.117124 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.117482 (+ 358us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.117900 (+ 418us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.119307 (+ 1407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.120615 (+ 1308us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.121486 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.121985 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.126245 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:57677: BlockingRecv error: recv got EOF from 127.6.115.65:57677 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
W20251210 14:58:54.127584 16180 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:40807 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.115.126:40807: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.132346 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.127964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:33041)
1210 14:58:54.128142 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.128156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.128238 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.128611 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.128615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.128760 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.128927 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.128932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.129666 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.129669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.130004 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.130010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.130106 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.130692 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.130704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.132224 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20251210 14:58:54.132462 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.128041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:33041 (local address 127.6.115.126:40807)
1210 14:58:54.128143 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.128159 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.128199 (+ 40us) server_negotiation.cc:408] Connection header received
1210 14:58:54.128469 (+ 270us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.128473 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.128513 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.128582 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.129039 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.129564 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.130104 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.130323 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.132336 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:33041: BlockingRecv error: recv got EOF from 127.6.115.65:33041 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.137275 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.132994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:56749)
1210 14:58:54.133157 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.133171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.133249 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.133633 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.133637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.133771 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.133922 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.133927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.134733 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.134736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.135067 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.135073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.135175 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.135650 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.135668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.137166 (+ 1498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20251210 14:58:54.137423 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.133064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:56749 (local address 127.6.115.126:40807)
1210 14:58:54.133170 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.133174 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.133213 (+ 39us) server_negotiation.cc:408] Connection header received
1210 14:58:54.133333 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.133336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.133506 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.133604 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.134127 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.134585 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.135206 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.135771 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.137295 (+ 1524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:56749: BlockingRecv error: recv got EOF from 127.6.115.65:56749 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20251210 14:58:54.137728 16180 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251210 14:58:54.231264 16184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:54.231446 16184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:54.231495 16184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:54.234364 16184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:54.234436 16184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:54.234508 16184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:58:54.238361 16184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.66.keytab
--principal=oryx/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:38037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=37209
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:54.239169 16184 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:54.239903 16184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:54.245702 16192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:54.245733 16190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:54.245891 16184 server_base.cc:1047] running on GCE node
W20251210 14:58:54.245954 16189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:54.246264 16184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:54.246742 16184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:54.247913 16184 hybrid_clock.cc:648] HybridClock initialized: now 1765378734247892 us; error 39 us; skew 500 ppm
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378734, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:54.250344 16184 init.cc:377] Logged in from keytab as oryx/127.6.115.66@KRBTEST.COM (short username oryx)
I20251210 14:58:54.251257 16184 webserver.cc:492] Webserver started at http://127.6.115.66:37209/ using document root <none> and password file <none>
I20251210 14:58:54.251718 16184 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:54.251776 16184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:54.254966 16184 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:58:54.256623 16199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.257581 16184 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:54.257666 16184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "92a4920d66284c0aa8237d2b9cc38bf3"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "70fdfa72d9414486562509c5fb5bd863"
server_key_iv: "ba2fdc6bc001b785bc6066337f31f857"
server_key_version: "encryptionkey@0"
I20251210 14:58:54.257997 16184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:54.270642 16184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:54.273464 16184 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:54.273615 16184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:54.274032 16184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:54.274747 16184 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:54.274799 16184 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.274843 16184 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:54.274859 16184 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.283354 16184 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:38037
I20251210 14:58:54.283392 16312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:38037 every 8 connection(s)
I20251210 14:58:54.284220 16184 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:58:54.287691 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16184
I20251210 14:58:54.288102 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:41985
--local_ip_for_outbound_sockets=127.6.115.67
--tserver_master_addrs=127.6.115.126:40807
--webserver_port=39187
--webserver_interface=127.6.115.67
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.115.67
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
W20251210 14:58:54.293701 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.285794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:60887 (local address 127.6.115.126:40807)
1210 14:58:54.285951 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.285956 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.286550 (+ 594us) server_negotiation.cc:408] Connection header received
1210 14:58:54.287254 (+ 704us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.287257 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.287318 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.287399 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.289025 (+ 1626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.289675 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.290404 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.290610 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.293535 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:60887: BlockingRecv error: recv got EOF from 127.6.115.66:60887 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251210 14:58:54.293977 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.286044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:60887)
1210 14:58:54.286424 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.286455 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.287080 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.287532 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.287538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.287907 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.288871 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.288886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.289803 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.289809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.290285 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.290294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.290469 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.290967 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.290990 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.293336 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
W20251210 14:58:54.294445 16313 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:40807 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.115.126:40807: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.299320 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.294857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:40731)
1210 14:58:54.295079 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.295090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.295172 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.295402 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.295404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.295521 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.295681 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.295686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.296679 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.296682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.297037 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.297043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.297142 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.297584 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.297595 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.299247 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167}
W20251210 14:58:54.299672 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.294975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:40731 (local address 127.6.115.126:40807)
1210 14:58:54.295119 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.295124 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.295147 (+ 23us) server_negotiation.cc:408] Connection header received
1210 14:58:54.295240 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.295246 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.295303 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.295394 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.295785 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.296568 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.297157 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.297320 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.299388 (+ 2068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:40731: BlockingRecv error: recv got EOF from 127.6.115.66:40731 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.304035 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.299790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:44425)
1210 14:58:54.300040 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.300050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.300108 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.300334 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.300336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.300464 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.300595 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.300600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.301636 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.301639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.301963 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.301969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.302054 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.302464 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.302474 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.303961 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196}
W20251210 14:58:54.304164 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.299941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:44425 (local address 127.6.115.126:40807)
1210 14:58:54.300067 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.300074 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.300092 (+ 18us) server_negotiation.cc:408] Connection header received
1210 14:58:54.300190 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.300193 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.300239 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.300321 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.300703 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.301536 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.302065 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.302245 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.304074 (+ 1829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:44425: BlockingRecv error: recv got EOF from 127.6.115.66:44425 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20251210 14:58:54.304364 16313 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251210 14:58:54.388834 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:58:54.388994 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:58:54.389038 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:58:54.392025 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:58:54.392068 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:58:54.392139 16316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:58:54.395905 16316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/krb5kdc/oryx_127.6.115.67.keytab
--principal=oryx/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:41985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=39187
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:58:54.396679 16316 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:58:54.397310 16316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:58:54.402686 16322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:58:54.402711 16324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:54.402925 16316 server_base.cc:1047] running on GCE node
W20251210 14:58:54.403040 16321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:58:54.403383 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:58:54.403859 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:58:54.405035 16316 hybrid_clock.cc:648] HybridClock initialized: now 1765378734405024 us; error 34 us; skew 500 ppm
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378734, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:58:54.407618 16316 init.cc:377] Logged in from keytab as oryx/127.6.115.67@KRBTEST.COM (short username oryx)
I20251210 14:58:54.408548 16316 webserver.cc:492] Webserver started at http://127.6.115.67:39187/ using document root <none> and password file <none>
I20251210 14:58:54.409009 16316 fs_manager.cc:362] Metadata directory not provided
I20251210 14:58:54.409072 16316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:58:54.412185 16316 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:54.413902 16331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.414804 16316 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:58:54.414891 16316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "133a4b8e05ae4652a1477951cdba6b00"
format_stamp: "Formatted at 2025-12-10 14:58:53 on dist-test-slave-nb51"
server_key: "7b56f9eea2c866435279c69fd3408d56"
server_key_iv: "b5584213576e5b16441aa049dcf4e07a"
server_key_version: "encryptionkey@0"
I20251210 14:58:54.415251 16316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:58:54.431522 16316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:58:54.434262 16316 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:58:54.434422 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:58:54.434873 16316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:58:54.435662 16316 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:58:54.435712 16316 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.435751 16316 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:58:54.435777 16316 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:58:54.444730 16316 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:41985
I20251210 14:58:54.444780 16444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:41985 every 8 connection(s)
I20251210 14:58:54.445652 16316 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.453256 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16316
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
W20251210 14:58:54.454808 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.447230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:40197 (local address 127.6.115.126:40807)
1210 14:58:54.447349 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.447353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.448010 (+ 657us) server_negotiation.cc:408] Connection header received
1210 14:58:54.448830 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.448835 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.448893 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.448966 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.450553 (+ 1587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.451018 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.451622 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.451790 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.454700 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:40197: BlockingRecv error: recv got EOF from 127.6.115.67:40197 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:58:54.455222 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.447468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:40197)
1210 14:58:54.447854 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.447885 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.448604 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.449123 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.449132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.449539 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.450407 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.450420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.451131 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.451134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.451514 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.451521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.451708 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.452296 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.452317 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.454587 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
W20251210 14:58:54.455613 16445 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:40807 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.115.126:40807: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.460505 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.456039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:36103)
1210 14:58:54.456143 (+ 104us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.456159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.456232 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.456570 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.456572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.456698 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.456853 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.456858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.457751 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.457754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.458086 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.458093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.458190 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.458634 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.458647 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.460431 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20251210 14:58:54.460698 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.456259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:36103 (local address 127.6.115.126:40807)
1210 14:58:54.456372 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.456375 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.456386 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:54.456413 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.456415 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.456458 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.456554 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.456961 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.457639 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.458213 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.458402 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.460577 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:36103: BlockingRecv error: recv got EOF from 127.6.115.67:36103 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20251210 14:58:54.462667 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.454151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.6.115.126:40807)
1210 14:58:54.454389 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.454393 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.454404 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:58:54.454431 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.454433 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.454467 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.454543 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.455718 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.456375 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.457121 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.457283 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.458512 (+ 1229us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:58:54.458536 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:58:54.458551 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:58:54.458603 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:58:54.460549 (+ 1946us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:54.461027 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:54.461035 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:54.461039 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:54.461158 (+ 119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:58:54.461406 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:58:54.461409 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:58:54.461411 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:58:54.461732 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:58:54.461928 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
1210 14:58:54.462339 (+ 411us) server_negotiation.cc:300] Negotiation successful
1210 14:58:54.462568 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:54 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:54.465830 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:54.461217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:46217)
1210 14:58:54.461421 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:54.461435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:54.461505 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:54.461830 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:54.461833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:54.461983 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:54.462142 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.462150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.463017 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.463021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:54.463522 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:54.463534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.463629 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.464044 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:54.464056 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:54.465715 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20251210 14:58:54.465921 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:54.461274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:46217 (local address 127.6.115.126:40807)
1210 14:58:54.461449 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:54.461453 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:54.461469 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:58:54.461647 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:54.461652 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:54.461695 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:54.461767 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:54.462322 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.462913 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:54.463639 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:54.463795 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:54.465823 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:46217: BlockingRecv error: recv got EOF from 127.6.115.67:46217 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20251210 14:58:54.466032 16445 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:55.143157 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:55.138255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:53535)
1210 14:58:55.138504 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:55.138516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:55.138601 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:55.138844 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:55.138846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:55.139013 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:55.139257 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.139263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.140003 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.140005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:55.140456 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.140463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.140550 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.141164 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:55.141176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:55.142988 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
W20251210 14:58:55.143258 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:55.138362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:53535 (local address 127.6.115.126:40807)
1210 14:58:55.138587 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:55.138591 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:55.138604 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:55.138692 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:55.138695 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:55.138734 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:55.138824 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:55.139392 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.139898 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.140560 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.140702 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.143124 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:53535: BlockingRecv error: recv got EOF from 127.6.115.65:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:55.309793 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:55.304838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:41349)
1210 14:58:55.305124 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:55.305141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:55.305330 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:55.305633 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:55.305637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:55.305859 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:55.306097 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.306104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.306816 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.306820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:55.307227 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.307234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.307360 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.307916 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:55.307928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:55.309652 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
W20251210 14:58:55.309854 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:55.304949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:41349 (local address 127.6.115.126:40807)
1210 14:58:55.305117 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:55.305121 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:55.305297 (+ 176us) server_negotiation.cc:408] Connection header received
1210 14:58:55.305481 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:55.305485 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:55.305533 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:55.305609 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:55.306220 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.306713 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.307340 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.307529 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.309762 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:41349: BlockingRecv error: recv got EOF from 127.6.115.66:41349 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:55 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:55.471627 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:55.466579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:44881)
1210 14:58:55.466860 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:55.466875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:55.466985 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:55.467237 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:55.467242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:55.467447 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:55.467757 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.467769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.468536 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.468539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:55.469053 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:55.469082 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.469204 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.469874 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:55.469890 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:55.471498 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
W20251210 14:58:55.471769 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:55.466875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:44881 (local address 127.6.115.126:40807)
1210 14:58:55.467000 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:55.467004 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:55.467070 (+ 66us) server_negotiation.cc:408] Connection header received
1210 14:58:55.467101 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:55.467103 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:55.467147 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:55.467212 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:55.467882 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.468441 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:55.469215 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:55.469477 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:55.471644 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:44881: BlockingRecv error: recv got EOF from 127.6.115.67:44881 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:56.148377 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:56.143817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:42585)
1210 14:58:56.144094 (+ 277us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:56.144107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:56.144191 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:56.144540 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:56.144544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:56.144722 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:56.144913 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.144919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.145653 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.145656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:56.146055 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.146061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.146144 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.146670 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:56.146681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:56.148281 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
W20251210 14:58:56.148499 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:56.143870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:42585 (local address 127.6.115.126:40807)
1210 14:58:56.144133 (+ 263us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:56.144136 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:56.144149 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:56.144309 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:56.144312 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:56.144358 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:56.144450 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:56.145030 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.145557 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.146137 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.146256 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.148392 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:42585: BlockingRecv error: recv got EOF from 127.6.115.65:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:56.315426 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:56.310474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:46611)
1210 14:58:56.310697 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:56.310709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:56.310801 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:56.311073 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:56.311076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:56.311244 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:56.311465 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.311470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.312202 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.312204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:56.312577 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.312583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.312690 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.313351 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:56.313391 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:56.315318 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
W20251210 14:58:56.315572 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:56.310577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:46611 (local address 127.6.115.126:40807)
1210 14:58:56.310697 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:56.310700 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:56.310767 (+ 67us) server_negotiation.cc:408] Connection header received
1210 14:58:56.310910 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:56.310913 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:56.310977 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:56.311046 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:56.311590 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.312102 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.312695 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.312832 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.315456 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:46611: BlockingRecv error: recv got EOF from 127.6.115.66:46611 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:56 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:56.477304 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:56.472276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:37491)
1210 14:58:56.472579 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:56.472597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:56.472738 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:56.473022 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:56.473025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:56.473228 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:56.473481 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.473489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.474195 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.474202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:56.474573 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:56.474581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.474693 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.475290 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:56.475309 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:56.477152 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
W20251210 14:58:56.477430 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:56.472459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:37491 (local address 127.6.115.126:40807)
1210 14:58:56.472585 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:56.472592 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:56.472686 (+ 94us) server_negotiation.cc:408] Connection header received
1210 14:58:56.472832 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:56.472837 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:56.472889 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:56.472964 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:56.473594 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.474092 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:56.474677 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:56.474884 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:56.477297 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:37491: BlockingRecv error: recv got EOF from 127.6.115.67:37491 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:57.157445 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:57.149091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:41703)
1210 14:58:57.149390 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:57.149404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:57.149489 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:57.149713 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:57.149716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:57.149899 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:57.150138 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.150144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.150775 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.150778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:57.151216 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.151221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.151302 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.155485 (+ 4183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:57.155499 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:57.157301 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
W20251210 14:58:57.157585 16459 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:57.149225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:41703 (local address 127.6.115.126:40807)
1210 14:58:57.149454 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:57.149457 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:57.149470 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:58:57.149566 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:57.149569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:57.149611 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:57.149689 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:57.150231 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.150708 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.151298 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.151418 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.157450 (+ 6032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:41703: BlockingRecv error: recv got EOF from 127.6.115.65:41703 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":62,"threads_started":1}
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:57.320883 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:57.316131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:55813)
1210 14:58:57.316418 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:57.316430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:57.316525 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:57.316755 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:57.316758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:57.316940 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:57.317131 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.317138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.317855 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.317858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:57.318242 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.318248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.318351 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.318935 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:57.318948 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:57.320712 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
W20251210 14:58:57.320940 16459 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:57.316195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:55813 (local address 127.6.115.126:40807)
1210 14:58:57.316338 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:57.316342 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:57.316512 (+ 170us) server_negotiation.cc:408] Connection header received
1210 14:58:57.316613 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:57.316615 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:57.316663 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:57.316725 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:57.317240 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.317764 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.318329 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.318506 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.320837 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:55813: BlockingRecv error: recv got EOF from 127.6.115.66:55813 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:57 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:57.483146 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:57.478029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:41777)
1210 14:58:57.478339 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:57.478353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:57.478469 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:57.478710 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:57.478713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:57.478926 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:57.479133 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.479140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.479974 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.479979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:57.480434 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:57.480443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.480557 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.481140 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:57.481154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:57.482969 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
W20251210 14:58:57.483193 16459 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:57.478093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:41777 (local address 127.6.115.126:40807)
1210 14:58:57.478256 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:57.478262 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:57.478425 (+ 163us) server_negotiation.cc:408] Connection header received
1210 14:58:57.478554 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:57.478559 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:57.478614 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:57.478686 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:57.479252 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.479868 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:57.480540 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:57.480752 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:57.483088 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:41777: BlockingRecv error: recv got EOF from 127.6.115.67:41777 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:58.162902 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:58.158155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:42267)
1210 14:58:58.158457 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:58.158470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:58.158567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:58.158797 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:58.158799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:58.158980 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:58.159187 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.159194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.159888 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.159891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:58.160287 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.160294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.160380 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.160928 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:58.160940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:58.162727 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
W20251210 14:58:58.162972 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:58.158228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:42267 (local address 127.6.115.126:40807)
1210 14:58:58.158478 (+ 250us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:58.158481 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:58.158520 (+ 39us) server_negotiation.cc:408] Connection header received
1210 14:58:58.158639 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:58.158641 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:58.158684 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:58.158757 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:58.159282 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.159790 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.160373 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.160500 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.162862 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:42267: BlockingRecv error: recv got EOF from 127.6.115.65:42267 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:58.326359 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:58.321598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:44149)
1210 14:58:58.321884 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:58.321895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:58.322007 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:58.322250 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:58.322253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:58.322451 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:58.322668 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.322676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.323523 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.323527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:58.323944 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.323952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.324051 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.324591 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:58.324603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:58.326254 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
W20251210 14:58:58.326464 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:58.321673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:44149 (local address 127.6.115.126:40807)
1210 14:58:58.321811 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:58.321816 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:58.321962 (+ 146us) server_negotiation.cc:408] Connection header received
1210 14:58:58.322095 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:58.322097 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:58.322138 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:58.322193 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:58.322764 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.323407 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.324073 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.324218 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.326364 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:44149: BlockingRecv error: recv got EOF from 127.6.115.66:44149 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:58 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:58.488343 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:58.483816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:38035)
1210 14:58:58.484110 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:58.484123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:58.484229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:58.484451 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:58.484454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:58.484641 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:58.484844 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.484849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.485584 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.485588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:58.485946 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:58.485952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.486055 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.486613 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:58.486624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:58.488176 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":114,"threads_started":1}
W20251210 14:58:58.488384 16463 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:58.483905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:38035 (local address 127.6.115.126:40807)
1210 14:58:58.484047 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:58.484051 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:58.484182 (+ 131us) server_negotiation.cc:408] Connection header received
1210 14:58:58.484326 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:58.484328 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:58.484369 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:58.484429 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:58.484960 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.485487 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:58.486052 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:58.486250 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:58.488298 (+ 2048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:38035: BlockingRecv error: recv got EOF from 127.6.115.67:38035 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:59.168210 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:59.163692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:60883)
1210 14:58:59.163947 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:59.163960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:59.164048 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:59.164274 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:59.164276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:59.164486 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:59.164676 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.164682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.165382 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.165385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:59.165749 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.165755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.165855 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.166399 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:59.166412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:59.168091 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
W20251210 14:58:59.168303 16467 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:59.163758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:60883 (local address 127.6.115.126:40807)
1210 14:58:59.163983 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:59.163987 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:59.164003 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:58:59.164127 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:59.164129 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:59.164172 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:59.164253 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:59.164772 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.165279 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.165858 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.165993 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.168202 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:60883: BlockingRecv error: recv got EOF from 127.6.115.65:60883 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:59.331902 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:59.327197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:43675)
1210 14:58:59.327463 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:59.327478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:59.327574 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:59.327842 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:59.327845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:59.328060 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:59.328246 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.328252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.328980 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.328984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:59.329354 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.329385 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.329492 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.330053 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:59.330066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:59.331753 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
W20251210 14:58:59.331990 16467 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:59.327416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:43675 (local address 127.6.115.126:40807)
1210 14:58:59.327529 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:59.327532 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:59.327544 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:58:59.327678 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:59.327681 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:59.327729 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:59.327806 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:59.328365 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.328876 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.329489 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.329722 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.331880 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:43675: BlockingRecv error: recv got EOF from 127.6.115.66:43675 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:58:59 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:58:59.493340 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:58:59.489045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:33729)
1210 14:58:59.489277 (+ 232us) negotiation.cc:107] Waiting for socket to connect
1210 14:58:59.489290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:58:59.489412 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:58:59.489628 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:58:59.489631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:58:59.489842 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:58:59.490019 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.490024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.490612 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.490614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:58:59.490960 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:58:59.490966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.491043 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.491526 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:58:59.491538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:58:59.493171 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
W20251210 14:58:59.493413 16467 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:58:59.489089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:33729 (local address 127.6.115.126:40807)
1210 14:58:59.489201 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1210 14:58:59.489204 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:58:59.489345 (+ 141us) server_negotiation.cc:408] Connection header received
1210 14:58:59.489498 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:58:59.489500 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:58:59.489542 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:58:59.489608 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:58:59.490111 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.490543 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:58:59.491028 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:58:59.491133 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:58:59.493287 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:33729: BlockingRecv error: recv got EOF from 127.6.115.67:33729 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:00.173890 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:00.168938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:50155)
1210 14:59:00.169224 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:00.169237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:00.169330 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:00.169575 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:00.169577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:00.169750 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:00.169972 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.169978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.170645 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.170648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:00.171145 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.171157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.171278 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.171964 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:00.171978 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:00.173783 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
W20251210 14:59:00.174046 16471 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:00.169053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:50155 (local address 127.6.115.126:40807)
1210 14:59:00.169278 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:00.169282 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:00.169295 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:59:00.169433 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:00.169436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:00.169479 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:00.169552 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:00.170056 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.170518 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.171285 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.171529 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.173904 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:50155: BlockingRecv error: recv got EOF from 127.6.115.65:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":55,"threads_started":1}
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:00.336858 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:00.332583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:59737)
1210 14:59:00.332807 (+ 224us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:00.332818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:00.332913 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:00.333120 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:00.333122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:00.333312 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:00.333516 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.333521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.334243 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.334246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:00.334616 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.334622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.334710 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.335232 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:00.335245 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:00.336757 (+ 1512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
W20251210 14:59:00.336961 16471 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:00.332659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:59737 (local address 127.6.115.126:40807)
1210 14:59:00.332807 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:00.332810 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:00.332876 (+ 66us) server_negotiation.cc:408] Connection header received
1210 14:59:00.332996 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:00.332998 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:00.333039 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:00.333100 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:00.333638 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.334147 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.334710 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.334840 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.336870 (+ 2030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:59737: BlockingRecv error: recv got EOF from 127.6.115.66:59737 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:00 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:00.498845 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:00.494127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:56065)
1210 14:59:00.494378 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:00.494391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:00.494496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:00.494737 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:00.494740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:00.494937 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:00.495120 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.495126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.495835 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.495838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:00.496238 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:00.496244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.496341 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.496873 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:00.496885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:00.498696 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
W20251210 14:59:00.498929 16471 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:00.494187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:56065 (local address 127.6.115.126:40807)
1210 14:59:00.494320 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:00.494324 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:00.494449 (+ 125us) server_negotiation.cc:408] Connection header received
1210 14:59:00.494577 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:00.494582 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:00.494626 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:00.494704 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:00.495226 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.495732 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:00.496334 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:00.496531 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:00.498818 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:56065: BlockingRecv error: recv got EOF from 127.6.115.67:56065 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:01.178854 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:01.174539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:42483)
1210 14:59:01.174863 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:01.174889 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:01.174986 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:01.175193 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:01.175195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:01.175367 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:01.175558 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.175564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.176235 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.176237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:01.176588 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.176593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.176675 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.177204 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:01.177216 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:01.178756 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
W20251210 14:59:01.178972 16475 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:01.174634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:42483 (local address 127.6.115.126:40807)
1210 14:59:01.174871 (+ 237us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:01.174874 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:01.174948 (+ 74us) server_negotiation.cc:408] Connection header received
1210 14:59:01.175052 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:01.175055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:01.175098 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:01.175175 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:01.175649 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.176163 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.176668 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.176795 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.178872 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:42483: BlockingRecv error: recv got EOF from 127.6.115.65:42483 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:01.342434 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:01.337446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:57729)
1210 14:59:01.337742 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:01.337757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:01.337892 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:01.338107 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:01.338110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:01.338282 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:01.338466 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.338472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.339155 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.339158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:01.339548 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.339555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.339658 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.340222 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:01.340236 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:01.342335 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
W20251210 14:59:01.342517 16475 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:01.337803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:57729 (local address 127.6.115.126:40807)
1210 14:59:01.337935 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:01.337938 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:01.337949 (+ 11us) server_negotiation.cc:408] Connection header received
1210 14:59:01.337978 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:01.337980 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:01.338019 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:01.338080 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:01.338579 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.339057 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.339637 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.339756 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.342430 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:57729: BlockingRecv error: recv got EOF from 127.6.115.66:57729 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:01 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:01.504328 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:01.499602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:33321)
1210 14:59:01.499894 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:01.499907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:01.500005 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:01.500218 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:01.500220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:01.500388 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:01.500586 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.500614 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.501353 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.501356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:01.501772 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:01.501779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.501868 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.502386 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:01.502398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:01.504221 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
W20251210 14:59:01.504451 16475 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:01.499687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:33321 (local address 127.6.115.126:40807)
1210 14:59:01.499804 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:01.499807 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:01.499974 (+ 167us) server_negotiation.cc:408] Connection header received
1210 14:59:01.500077 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:01.500080 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:01.500129 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:01.500196 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:01.500722 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.501229 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:01.501867 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:01.501998 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:01.504348 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:33321: BlockingRecv error: recv got EOF from 127.6.115.67:33321 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:02.187297 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:02.179634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:47425)
1210 14:59:02.179965 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:02.179982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:02.180082 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:02.180366 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:02.180369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:02.180637 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:02.180892 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.180901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.181686 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.181690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:02.182099 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.182106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.182206 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.185431 (+ 3225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:02.185444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:02.187135 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
W20251210 14:59:02.187369 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:02.179754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:47425 (local address 127.6.115.126:40807)
1210 14:59:02.180017 (+ 263us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:02.180020 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:02.180036 (+ 16us) server_negotiation.cc:408] Connection header received
1210 14:59:02.180193 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:02.180197 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:02.180242 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:02.180334 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:02.181015 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.181593 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.182198 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.182406 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.187253 (+ 4847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:47425: BlockingRecv error: recv got EOF from 127.6.115.65:47425 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:02.347844 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:02.343122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:45753)
1210 14:59:02.343355 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:02.343366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:02.343455 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:02.343678 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:02.343680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:02.343842 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:02.344038 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.344045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.345032 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.345035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:02.345438 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.345445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.345534 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.346113 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:02.346126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:02.347731 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
W20251210 14:59:02.348045 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:02.343183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:45753 (local address 127.6.115.126:40807)
1210 14:59:02.343319 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:02.343322 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:02.343446 (+ 124us) server_negotiation.cc:408] Connection header received
1210 14:59:02.343548 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:02.343551 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:02.343595 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:02.343658 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:02.344173 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.344919 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.345525 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.345674 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.347897 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:45753: BlockingRecv error: recv got EOF from 127.6.115.66:45753 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:02 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:02.509251 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:02.505057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:34891)
1210 14:59:02.505321 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:02.505334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:02.505451 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:02.505684 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:02.505687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:02.505859 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:02.506048 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.506053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.506680 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.506682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:02.507046 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:02.507051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.507129 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.507621 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:02.507633 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:02.509146 (+ 1513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20251210 14:59:02.509394 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:02.505140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:34891 (local address 127.6.115.126:40807)
1210 14:59:02.505282 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:02.505287 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:02.505410 (+ 123us) server_negotiation.cc:408] Connection header received
1210 14:59:02.505542 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:02.505545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:02.505600 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:02.505660 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:02.506138 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.506611 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:02.507124 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:02.507239 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:02.509257 (+ 2018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:34891: BlockingRecv error: recv got EOF from 127.6.115.67:34891 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:03.192544 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:03.188020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:58743)
1210 14:59:03.188268 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:03.188281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:03.188375 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:03.188593 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:03.188595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:03.188794 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:03.188973 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.188979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.189667 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.189669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:03.190055 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.190061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.190146 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.190675 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:03.190687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:03.192392 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
W20251210 14:59:03.192641 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:03.188081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:58743 (local address 127.6.115.126:40807)
1210 14:59:03.188291 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:03.188295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:03.188329 (+ 34us) server_negotiation.cc:408] Connection header received
1210 14:59:03.188447 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:03.188450 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:03.188492 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:03.188573 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:03.189061 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.189584 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.190141 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.190267 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.192519 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:58743: BlockingRecv error: recv got EOF from 127.6.115.65:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:03.354106 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:03.348981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:44499)
1210 14:59:03.349227 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:03.349240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:03.349335 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:03.349607 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:03.349609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:03.349833 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:03.350086 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.350096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.350946 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.350948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:03.351299 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.351305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.351388 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.351942 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:03.351954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:03.354001 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
W20251210 14:59:03.354282 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:03.349037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:44499 (local address 127.6.115.126:40807)
1210 14:59:03.349175 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:03.349179 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:03.349301 (+ 122us) server_negotiation.cc:408] Connection header received
1210 14:59:03.349444 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:03.349447 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:03.349506 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:03.349592 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:03.350243 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.350840 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.351391 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.351533 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.354135 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:44499: BlockingRecv error: recv got EOF from 127.6.115.66:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:03 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:03.514885 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:03.510209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:52349)
1210 14:59:03.510599 (+ 390us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:03.510616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:03.510731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:03.510998 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:03.511000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:03.511184 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:03.511364 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.511369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.512021 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.512024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:03.512412 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:03.512418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.512506 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.513004 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:03.513033 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:03.514704 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
W20251210 14:59:03.514940 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:03.510247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:52349 (local address 127.6.115.126:40807)
1210 14:59:03.510433 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:03.510454 (+ 21us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:03.510689 (+ 235us) server_negotiation.cc:408] Connection header received
1210 14:59:03.510840 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:03.510843 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:03.510894 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:03.510958 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:03.511457 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.511920 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:03.512505 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:03.512633 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:03.514835 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:52349: BlockingRecv error: recv got EOF from 127.6.115.67:52349 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:04.198488 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:04.193416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:49033)
1210 14:59:04.193683 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:04.193696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:04.193775 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:04.193988 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:04.193990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:04.194161 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:04.194339 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.194344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.195110 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.195115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:04.195642 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.195652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.195787 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.196466 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:04.196478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:04.198330 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
W20251210 14:59:04.198704 16487 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:04.193496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:49033 (local address 127.6.115.126:40807)
1210 14:59:04.193753 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:04.193756 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:04.193770 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:59:04.193848 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:04.193850 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:04.193889 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:04.193965 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:04.194439 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.194992 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.195776 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.195993 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.198472 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:49033: BlockingRecv error: recv got EOF from 127.6.115.65:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:04.359182 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:04.354767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:36515)
1210 14:59:04.355018 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:04.355031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:04.355124 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:04.355343 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:04.355345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:04.355517 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:04.355710 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.355716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.356391 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.356394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:04.356772 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.356777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.356858 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.357354 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:04.357391 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:04.359024 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
W20251210 14:59:04.359238 16487 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:04.354840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:36515 (local address 127.6.115.126:40807)
1210 14:59:04.355017 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:04.355021 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:04.355093 (+ 72us) server_negotiation.cc:408] Connection header received
1210 14:59:04.355205 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:04.355208 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:04.355252 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:04.355310 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:04.355821 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.356319 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.356862 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.356990 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.359138 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:36515: BlockingRecv error: recv got EOF from 127.6.115.66:36515 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:04 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:04.520195 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:04.515633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:51489)
1210 14:59:04.515894 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:04.515907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:04.516017 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:04.516230 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:04.516232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:04.516392 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:04.516591 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.516597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.517279 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.517282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:04.517711 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:04.517718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.517805 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.518310 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:04.518323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:04.520061 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
W20251210 14:59:04.520334 16487 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:04.515771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:51489 (local address 127.6.115.126:40807)
1210 14:59:04.515932 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:04.515935 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:04.515965 (+ 30us) server_negotiation.cc:408] Connection header received
1210 14:59:04.516095 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:04.516098 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:04.516150 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:04.516212 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:04.516707 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.517179 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:04.517827 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:04.517977 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:04.520203 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:51489: BlockingRecv error: recv got EOF from 127.6.115.67:51489 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:05.204082 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:05.199201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:33741)
1210 14:59:05.199498 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:05.199512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:05.199603 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:05.199825 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:05.199827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:05.200006 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:05.200195 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.200201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.200916 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.200919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:05.201298 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.201304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.201452 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.202028 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:05.202041 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:05.203931 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
W20251210 14:59:05.204269 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:05.199315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:33741 (local address 127.6.115.126:40807)
1210 14:59:05.199554 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:05.199558 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:05.199571 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:59:05.199681 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:05.199684 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:05.199727 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:05.199798 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:05.200298 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.200824 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.201424 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.201567 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.204105 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:33741: BlockingRecv error: recv got EOF from 127.6.115.65:33741 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:05.364801 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:05.359913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:57637)
1210 14:59:05.360167 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:05.360179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:05.360288 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:05.360507 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:05.360509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:05.360680 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:05.360879 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.360885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.361761 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.361766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:05.362393 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.362403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.362493 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.363011 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:05.363023 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:05.364692 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
W20251210 14:59:05.364903 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:05.359972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:57637 (local address 127.6.115.126:40807)
1210 14:59:05.360095 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:05.360098 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:05.360253 (+ 155us) server_negotiation.cc:408] Connection header received
1210 14:59:05.360373 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:05.360376 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:05.360422 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:05.360483 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:05.360971 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.361651 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.362508 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.362652 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.364803 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:57637: BlockingRecv error: recv got EOF from 127.6.115.66:57637 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:05 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:05.525147 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:05.520977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:36619)
1210 14:59:05.521211 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:05.521224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:05.521320 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:05.521567 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:05.521569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:05.521741 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:05.521931 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.521937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.522596 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.522598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:05.522967 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:05.522973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.523050 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.523562 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:05.523572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:05.525043 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
W20251210 14:59:05.525245 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:05.521055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:36619 (local address 127.6.115.126:40807)
1210 14:59:05.521177 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:05.521180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:05.521280 (+ 100us) server_negotiation.cc:408] Connection header received
1210 14:59:05.521413 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:05.521416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:05.521461 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:05.521535 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:05.522025 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.522499 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:05.523049 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:05.523169 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:05.525153 (+ 1984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:36619: BlockingRecv error: recv got EOF from 127.6.115.67:36619 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:06.209134 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:06.204695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:39187)
1210 14:59:06.204952 (+ 257us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:06.204965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:06.205052 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:06.205269 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:06.205271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:06.205464 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:06.205666 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.205673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.206374 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.206377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:06.206744 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.206750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.206833 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.207365 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:06.207378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:06.209034 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
W20251210 14:59:06.209291 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:06.204773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:39187 (local address 127.6.115.126:40807)
1210 14:59:06.205028 (+ 255us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:06.205032 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:06.205044 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:59:06.205124 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:06.205126 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:06.205168 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:06.205241 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:06.205766 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.206267 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.206824 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.206968 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.209155 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:39187: BlockingRecv error: recv got EOF from 127.6.115.65:39187 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:06.370173 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:06.365491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:34803)
1210 14:59:06.365764 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:06.365777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:06.365874 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:06.366080 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:06.366082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:06.366251 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:06.366461 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.366467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.367209 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.367211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:06.367601 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.367608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.367696 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.368242 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:06.368255 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:06.370064 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20251210 14:59:06.370287 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:06.365567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:34803 (local address 127.6.115.126:40807)
1210 14:59:06.365698 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:06.365701 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:06.365838 (+ 137us) server_negotiation.cc:408] Connection header received
1210 14:59:06.365944 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:06.365947 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:06.365989 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:06.366057 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:06.366601 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.367113 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.367696 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.367831 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.370184 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:34803: BlockingRecv error: recv got EOF from 127.6.115.66:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:06 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:06.531076 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:06.525857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:56677)
1210 14:59:06.526165 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:06.526176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:06.526277 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:06.526682 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:06.526685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:06.526888 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:06.527078 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.527085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.527874 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.527877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:06.528245 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:06.528252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.528368 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.528945 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:06.528958 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:06.530935 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
W20251210 14:59:06.531167 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:06.525907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:56677 (local address 127.6.115.126:40807)
1210 14:59:06.526083 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:06.526087 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:06.526238 (+ 151us) server_negotiation.cc:408] Connection header received
1210 14:59:06.526381 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:06.526385 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:06.526537 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:06.526645 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:06.527222 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.527738 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:06.528364 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:06.528568 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:06.531061 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:56677: BlockingRecv error: recv got EOF from 127.6.115.67:56677 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:07.218470 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:07.209859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:49569)
1210 14:59:07.210118 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:07.210132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:07.210237 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:07.210497 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:07.210499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:07.210672 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:07.210890 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.210896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.211694 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.211698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:07.212078 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.212085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.212178 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.216511 (+ 4333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:07.216524 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:07.218366 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
W20251210 14:59:07.218665 16499 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:07.209931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:49569 (local address 127.6.115.126:40807)
1210 14:59:07.210151 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:07.210154 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:07.210180 (+ 26us) server_negotiation.cc:408] Connection header received
1210 14:59:07.210347 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:07.210350 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:07.210392 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:07.210465 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:07.210994 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.211576 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.212168 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.212305 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.218521 (+ 6216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:49569: BlockingRecv error: recv got EOF from 127.6.115.65:49569 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:07.375700 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:07.370858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:49067)
1210 14:59:07.371113 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:07.371125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:07.371217 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:07.371427 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:07.371429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:07.371603 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:07.371782 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.371787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.372456 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.372459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:07.372863 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.372871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.372961 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.373557 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:07.373570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:07.375538 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
W20251210 14:59:07.375782 16499 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:07.370999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:49067 (local address 127.6.115.126:40807)
1210 14:59:07.371116 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:07.371119 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:07.371184 (+ 65us) server_negotiation.cc:408] Connection header received
1210 14:59:07.371286 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:07.371288 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:07.371330 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:07.371391 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:07.371874 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.372355 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.372962 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.373106 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.375668 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:49067: BlockingRecv error: recv got EOF from 127.6.115.66:49067 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:07 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:07.536571 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:07.531732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:42835)
1210 14:59:07.532022 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:07.532038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:07.532157 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:07.532459 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:07.532463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:07.532687 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:07.532955 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.532968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.533733 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.533735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:07.534098 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:07.534104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.534190 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.534728 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:07.534741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:07.536461 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20251210 14:59:07.536706 16499 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:07.531823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:42835 (local address 127.6.115.126:40807)
1210 14:59:07.531990 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:07.531994 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:07.532114 (+ 120us) server_negotiation.cc:408] Connection header received
1210 14:59:07.532258 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:07.532261 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:07.532322 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:07.532401 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:07.533082 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.533630 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:07.534182 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:07.534313 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:07.536593 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:42835: BlockingRecv error: recv got EOF from 127.6.115.67:42835 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:08.223948 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:08.219162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:46943)
1210 14:59:08.219432 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:08.219449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:08.219548 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:08.219782 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:08.219784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:08.219951 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:08.220198 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.220205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.221022 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.221025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:08.221418 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.221424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.221515 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.222033 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:08.222050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:08.223844 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
W20251210 14:59:08.224112 16503 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:08.219242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:46943 (local address 127.6.115.126:40807)
1210 14:59:08.219528 (+ 286us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:08.219531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:08.219546 (+ 15us) server_negotiation.cc:408] Connection header received
1210 14:59:08.219630 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:08.219633 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:08.219680 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:08.219765 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:08.220350 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.220926 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.221504 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.221640 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.223971 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:46943: BlockingRecv error: recv got EOF from 127.6.115.65:46943 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:08.382081 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:08.376508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:41731)
1210 14:59:08.376790 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:08.376802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:08.376920 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:08.377160 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:08.377163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:08.377351 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:08.377572 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.377579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.378239 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.378242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:08.378597 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.378603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.378699 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.379540 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:08.379553 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:08.381929 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
W20251210 14:59:08.382125 16503 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:08.376559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:41731 (local address 127.6.115.126:40807)
1210 14:59:08.376725 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:08.376730 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:08.376870 (+ 140us) server_negotiation.cc:408] Connection header received
1210 14:59:08.377017 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:08.377020 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:08.377065 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:08.377130 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:08.377686 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.378143 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.378693 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.378908 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.382034 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:41731: BlockingRecv error: recv got EOF from 127.6.115.66:41731 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:08 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.67@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:08.542277 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:08.537283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.67:40959)
1210 14:59:08.537622 (+ 339us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:08.537637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:08.537757 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:08.538009 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:08.538012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:08.538211 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:08.538402 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.538408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.539160 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.539166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:08.539634 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:08.539643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.539748 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.540307 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:08.540319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:08.542129 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":146,"threads_started":1}
W20251210 14:59:08.542377 16503 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:08.537415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:40959 (local address 127.6.115.126:40807)
1210 14:59:08.537598 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:08.537603 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:08.537707 (+ 104us) server_negotiation.cc:408] Connection header received
1210 14:59:08.537858 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:08.537861 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:08.537911 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:08.537978 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:08.538519 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.539061 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:08.539744 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:08.539940 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:08.542274 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.67:40959: BlockingRecv error: recv got EOF from 127.6.115.67:40959 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Dec 10 14:59:09 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:09 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.65@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:09.229163 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:09.224682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.65:53875)
1210 14:59:09.224943 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:09.224955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:09.225052 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:09.225281 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:09.225283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:09.225484 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:09.225675 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.225681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.226363 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.226366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:09.226718 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.226736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.226816 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.227415 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:09.227428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:09.229060 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
W20251210 14:59:09.229346 16507 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:09.224753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:53875 (local address 127.6.115.126:40807)
1210 14:59:09.224962 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:09.224965 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:09.225005 (+ 40us) server_negotiation.cc:408] Connection header received
1210 14:59:09.225123 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:09.225125 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:09.225168 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:09.225247 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:09.225768 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.226282 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.226805 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.226946 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.229195 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.65:53875: BlockingRecv error: recv got EOF from 127.6.115.65:53875 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
Dec 10 14:59:09 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
Dec 10 14:59:09 dist-test-slave-nb51 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.115.66@KRBTEST.COM for oryx/127.6.115.126@KRBTEST.COM, Server not found in Kerberos database
I20251210 14:59:09.388051 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:09.382973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40807 (local address 127.6.115.66:56679)
1210 14:59:09.383234 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:09.383246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:09.383340 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:09.383607 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:09.383610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:09.383786 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:09.383979 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.383984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.384674 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.384677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:09.385075 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.385081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.385176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.385909 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:09.385924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:09.387952 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.115.126:40807: Server oryx/127.6.115.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
W20251210 14:59:09.388147 16507 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:09.383043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:56679 (local address 127.6.115.126:40807)
1210 14:59:09.383193 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:09.383197 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:09.383309 (+ 112us) server_negotiation.cc:408] Connection header received
1210 14:59:09.383409 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:09.383411 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:09.383462 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:09.383526 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:09.384071 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.384563 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.385157 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.385279 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.388061 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.115.66:56679: BlockingRecv error: recv got EOF from 127.6.115.66:56679 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20251210 14:59:09.464495 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16032
I20251210 14:59:09.471886 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16184
I20251210 14:59:09.478122 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16316
I20251210 14:59:09.485194 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 15962
2025-12-10T14:59:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18736 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-12-10T14:59:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:59:09Z Disabled control of system clock
I20251210 14:59:09.515120 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40073
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:40073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251210 14:59:09.611025 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:09.611215 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:09.611284 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:09.614051 16512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:59:09.614099 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:09.614125 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:09.614187 16512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:59:09.614223 16512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:59:09.617615 16512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:40073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:09.618422 16512 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:09.619345 16512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:09.624584 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.624605 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.624626 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:09.624682 16512 server_base.cc:1047] running on GCE node
I20251210 14:59:09.625249 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:09.626058 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:09.627202 16512 hybrid_clock.cc:648] HybridClock initialized: now 1765378749627194 us; error 23 us; skew 500 ppm
I20251210 14:59:09.628726 16512 webserver.cc:492] Webserver started at http://127.6.115.126:35357/ using document root <none> and password file <none>
I20251210 14:59:09.629227 16512 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:09.629280 16512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:09.629513 16512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:09.631021 16512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "ad7f926f555c4b529a41c591bc08f503"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "ed8e6714a84f4284ed7414fc19c3c4b2"
server_key_iv: "bf33b66560dc28b0b9d57cea5793da35"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.631426 16512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "ad7f926f555c4b529a41c591bc08f503"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "ed8e6714a84f4284ed7414fc19c3c4b2"
server_key_iv: "bf33b66560dc28b0b9d57cea5793da35"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.634346 16512 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:59:09.636276 16526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:09.637154 16512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:09.637257 16512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "ad7f926f555c4b529a41c591bc08f503"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "ed8e6714a84f4284ed7414fc19c3c4b2"
server_key_iv: "bf33b66560dc28b0b9d57cea5793da35"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.637331 16512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:09.656242 16512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:09.656774 16512 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:09.656925 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:09.663475 16512 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:40073
I20251210 14:59:09.663488 16578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:40073 every 8 connection(s)
I20251210 14:59:09.664358 16512 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:59:09.667042 16579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:09.669091 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16512
I20251210 14:59:09.669225 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:59:09.669579 6605 external_mini_cluster.cc:1468] Setting key c7a44d3e826568aec75e3ed633e9ee98
I20251210 14:59:09.672515 16579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503: Bootstrap starting.
I20251210 14:59:09.674409 16579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:09.675124 16579 log.cc:826] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:09.676469 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:09.670740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.6.115.126:40073)
1210 14:59:09.671292 (+ 552us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:09.671300 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:09.671329 (+ 29us) server_negotiation.cc:408] Connection header received
1210 14:59:09.671787 (+ 458us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:09.671799 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:09.672117 (+ 318us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:09.672405 (+ 288us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:09.672877 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.673590 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.674188 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.674434 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.674910 (+ 476us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:09.674929 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:09.674938 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:09.675324 (+ 386us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:09.675360 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:09.675374 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:09.675467 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:09.675620 (+ 153us) server_negotiation.cc:300] Negotiation successful
1210 14:59:09.675828 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20251210 14:59:09.676924 16579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503: No bootstrap required, opened a new log
I20251210 14:59:09.679327 16579 raft_consensus.cc:359] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } }
I20251210 14:59:09.679490 16579 raft_consensus.cc:385] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:09.679524 16579 raft_consensus.cc:740] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7f926f555c4b529a41c591bc08f503, State: Initialized, Role: FOLLOWER
I20251210 14:59:09.679850 16579 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } }
I20251210 14:59:09.679955 16579 raft_consensus.cc:399] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:59:09.680006 16579 raft_consensus.cc:493] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:59:09.680068 16579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:09.680727 16579 raft_consensus.cc:515] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } }
I20251210 14:59:09.680989 16579 leader_election.cc:304] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7f926f555c4b529a41c591bc08f503; no voters:
I20251210 14:59:09.681209 16579 leader_election.cc:290] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:59:09.681324 16584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:09.681559 16584 raft_consensus.cc:697] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [term 1 LEADER]: Becoming Leader. State: Replica: ad7f926f555c4b529a41c591bc08f503, State: Running, Role: LEADER
I20251210 14:59:09.681864 16584 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } }
I20251210 14:59:09.682158 16579 sys_catalog.cc:565] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:59:09.683156 16585 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad7f926f555c4b529a41c591bc08f503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } } }
I20251210 14:59:09.683279 16585 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:09.683285 16586 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7f926f555c4b529a41c591bc08f503. Latest consensus state: current_term: 1 leader_uuid: "ad7f926f555c4b529a41c591bc08f503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7f926f555c4b529a41c591bc08f503" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 40073 } } }
I20251210 14:59:09.683508 16586 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:09.683790 16593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:59:09.686578 16593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:59:09.691298 16593 catalog_manager.cc:1357] Generated new cluster ID: 584e777b430a474dadce45b5cb665278
I20251210 14:59:09.691359 16593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:59:09.700821 16593 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:59:09.701875 16593 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:59:09.711390 16593 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ad7f926f555c4b529a41c591bc08f503: Generated new TSK 0
I20251210 14:59:09.712033 16593 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:59:09.721668 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--builtin_ntp_servers=127.6.115.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251210 14:59:09.817786 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:09.817950 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:09.817986 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:09.821033 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:09.821079 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:09.821156 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:59:09.824661 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:09.825562 16603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:09.826548 16603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:09.831979 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.831987 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.832221 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:09.832248 16603 server_base.cc:1047] running on GCE node
I20251210 14:59:09.832618 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:09.833151 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:09.834301 16603 hybrid_clock.cc:648] HybridClock initialized: now 1765378749834269 us; error 48 us; skew 500 ppm
I20251210 14:59:09.836293 16603 webserver.cc:492] Webserver started at http://127.6.115.65:37111/ using document root <none> and password file <none>
I20251210 14:59:09.836927 16603 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:09.836997 16603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:09.837185 16603 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:09.839080 16603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "3663cdd13e704ae7bd565cd51c3c2447"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "7a08fdedace6f7052d613441182e7279"
server_key_iv: "3e0788b4f4eb2d98c1806839449d0fe8"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.839643 16603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "3663cdd13e704ae7bd565cd51c3c2447"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "7a08fdedace6f7052d613441182e7279"
server_key_iv: "3e0788b4f4eb2d98c1806839449d0fe8"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.842962 16603 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:59:09.844928 16617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:09.845826 16603 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:09.845920 16603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "3663cdd13e704ae7bd565cd51c3c2447"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "7a08fdedace6f7052d613441182e7279"
server_key_iv: "3e0788b4f4eb2d98c1806839449d0fe8"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.845999 16603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:09.864121 16603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:09.864571 16603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:09.864696 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:09.865121 16603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:09.866010 16603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:09.866055 16603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:09.866102 16603 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:09.866137 16603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:09.874100 16603 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:39365
I20251210 14:59:09.874122 16730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:39365 every 8 connection(s)
I20251210 14:59:09.874910 16603 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:59:09.876458 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16603
I20251210 14:59:09.876569 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:59:09.876837 6605 external_mini_cluster.cc:1468] Setting key 5022d7c786ccdd2f074b1e6b32045853
I20251210 14:59:09.878607 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--builtin_ntp_servers=127.6.115.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251210 14:59:09.882611 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:09.876412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:52183 (local address 127.6.115.126:40073)
1210 14:59:09.876530 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:09.876535 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:09.877564 (+ 1029us) server_negotiation.cc:408] Connection header received
1210 14:59:09.878375 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:09.878378 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:09.878443 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:09.878517 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:09.879462 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.879946 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.880583 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.880771 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.882006 (+ 1235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:09.882027 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:09.882033 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:09.882049 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:09.882075 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:09.882087 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:09.882169 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:09.882338 (+ 169us) server_negotiation.cc:300] Negotiation successful
1210 14:59:09.882416 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251210 14:59:09.883035 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:09.876771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40073 (local address 127.6.115.65:52183)
1210 14:59:09.877447 (+ 676us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:09.877481 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:09.878201 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:09.878673 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:09.878685 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:09.878788 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:09.879315 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.879325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.880083 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:09.880086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:09.880467 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:09.880473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:09.880668 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:09.881788 (+ 1120us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:09.881790 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:09.881816 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:09.881862 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:09.882191 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:09.882198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:09.882201 (+ 3us) client_negotiation.cc:769] Sending connection context
1210 14:59:09.882305 (+ 104us) client_negotiation.cc:240] Negotiation successful
1210 14:59:09.882467 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20251210 14:59:09.884153 16731 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40073
I20251210 14:59:09.884357 16731 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:09.884792 16731 heartbeater.cc:507] Master 127.6.115.126:40073 requested a full tablet report, sending...
I20251210 14:59:09.886090 16543 ts_manager.cc:194] Registered new tserver with Master: 3663cdd13e704ae7bd565cd51c3c2447 (127.6.115.65:39365)
I20251210 14:59:09.887203 16543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.115.65:52183
W20251210 14:59:09.976997 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:09.977180 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:09.977218 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:09.980185 16734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:09.980235 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:09.980306 16734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:59:09.984015 16734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:09.985005 16734 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:09.986042 16734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:09.991734 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.991739 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:09.991745 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:09.991930 16734 server_base.cc:1047] running on GCE node
I20251210 14:59:09.992329 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:09.992810 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:09.993978 16734 hybrid_clock.cc:648] HybridClock initialized: now 1765378749993961 us; error 34 us; skew 500 ppm
I20251210 14:59:09.995821 16734 webserver.cc:492] Webserver started at http://127.6.115.66:41559/ using document root <none> and password file <none>
I20251210 14:59:09.996387 16734 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:09.996454 16734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:09.996639 16734 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:09.998333 16734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "e7db30093c354ba693b6ecba0c983722"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "860aac3df77310acd864fe8a73efdbdc"
server_key_iv: "cab76351572a6a4ad0ef8bfd0bc04f5e"
server_key_version: "encryptionkey@0"
I20251210 14:59:09.998826 16734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "e7db30093c354ba693b6ecba0c983722"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "860aac3df77310acd864fe8a73efdbdc"
server_key_iv: "cab76351572a6a4ad0ef8bfd0bc04f5e"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.002148 16734 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251210 14:59:10.004351 16748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.005326 16734 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251210 14:59:10.005443 16734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "e7db30093c354ba693b6ecba0c983722"
format_stamp: "Formatted at 2025-12-10 14:59:09 on dist-test-slave-nb51"
server_key: "860aac3df77310acd864fe8a73efdbdc"
server_key_iv: "cab76351572a6a4ad0ef8bfd0bc04f5e"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.005529 16734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.016141 16734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.016639 16734 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.016799 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.017311 16734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:10.018234 16734 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:10.018287 16734 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.018349 16734 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:10.018386 16734 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.027529 16734 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:33087
I20251210 14:59:10.027555 16861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:33087 every 8 connection(s)
I20251210 14:59:10.028442 16734 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:59:10.033979 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16734
I20251210 14:59:10.034098 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:59:10.034369 6605 external_mini_cluster.cc:1468] Setting key ac208617dd593a86f24ed4a059c5f1f6
I20251210 14:59:10.036279 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.030130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:45999 (local address 127.6.115.126:40073)
1210 14:59:10.030298 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.030304 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.031257 (+ 953us) server_negotiation.cc:408] Connection header received
1210 14:59:10.032028 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.032031 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.032102 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.032187 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.033042 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.033646 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:10.034337 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.034538 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:10.035814 (+ 1276us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.035828 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.035830 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.035842 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.035864 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.035873 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.035946 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.036118 (+ 172us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.036172 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251210 14:59:10.036389 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--builtin_ntp_servers=127.6.115.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251210 14:59:10.037101 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.030407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40073 (local address 127.6.115.66:45999)
1210 14:59:10.031079 (+ 672us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:10.031119 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:10.031837 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:10.032319 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:10.032326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:10.032400 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:10.032921 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:10.032931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.033755 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:10.033758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:10.034222 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:10.034230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.034397 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:10.035564 (+ 1167us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:10.035569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:10.035608 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:10.035670 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:10.035965 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:10.035973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:10.035977 (+ 4us) client_negotiation.cc:769] Sending connection context
1210 14:59:10.036125 (+ 148us) client_negotiation.cc:240] Negotiation successful
1210 14:59:10.036342 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20251210 14:59:10.038081 16862 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40073
I20251210 14:59:10.038336 16862 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:10.038929 16862 heartbeater.cc:507] Master 127.6.115.126:40073 requested a full tablet report, sending...
I20251210 14:59:10.039958 16543 ts_manager.cc:194] Registered new tserver with Master: e7db30093c354ba693b6ecba0c983722 (127.6.115.66:33087)
I20251210 14:59:10.040516 16543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.115.66:45999
W20251210 14:59:10.138921 16865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:10.139138 16865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:10.139214 16865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:10.142539 16865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:10.142589 16865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:10.142701 16865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:59:10.146065 16865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:10.147030 16865 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:10.147991 16865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:10.154093 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.154187 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.154222 16865 server_base.cc:1047] running on GCE node
W20251210 14:59:10.154148 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.154670 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:10.155192 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:10.156396 16865 hybrid_clock.cc:648] HybridClock initialized: now 1765378750156376 us; error 39 us; skew 500 ppm
I20251210 14:59:10.158026 16865 webserver.cc:492] Webserver started at http://127.6.115.67:33087/ using document root <none> and password file <none>
I20251210 14:59:10.158543 16865 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:10.158608 16865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:10.158797 16865 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:10.160393 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "f7a23f48d31e4a9da447a1bc1166327f"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "77db1bd19aaa7924614c3508fe7dc311"
server_key_iv: "bdee49f75e08f99fb512619928ac005b"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.160853 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a23f48d31e4a9da447a1bc1166327f"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "77db1bd19aaa7924614c3508fe7dc311"
server_key_iv: "bdee49f75e08f99fb512619928ac005b"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.163981 16865 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:59:10.166074 16879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.166934 16865 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:10.167028 16865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "f7a23f48d31e4a9da447a1bc1166327f"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "77db1bd19aaa7924614c3508fe7dc311"
server_key_iv: "bdee49f75e08f99fb512619928ac005b"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.167133 16865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.188525 16865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.188961 16865 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.189095 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.189597 16865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:10.190371 16865 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:10.190414 16865 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.190464 16865 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:10.190491 16865 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.198591 16865 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:33029
I20251210 14:59:10.198618 16992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:33029 every 8 connection(s)
I20251210 14:59:10.199411 16865 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:59:10.201824 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 16865
I20251210 14:59:10.201960 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:59:10.202243 6605 external_mini_cluster.cc:1468] Setting key 5df131fbb080530e4b661f22d457e93b
I20251210 14:59:10.207144 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.200922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:58773 (local address 127.6.115.126:40073)
1210 14:59:10.201087 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.201093 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.201996 (+ 903us) server_negotiation.cc:408] Connection header received
1210 14:59:10.202948 (+ 952us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.202952 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.203019 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.203117 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.204060 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.204600 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:10.205217 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.205358 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:10.206656 (+ 1298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.206671 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.206673 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.206685 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.206705 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.206713 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.206784 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.206973 (+ 189us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.207020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251210 14:59:10.207649 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.201204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:40073 (local address 127.6.115.67:58773)
1210 14:59:10.201835 (+ 631us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:10.201892 (+ 57us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:10.202759 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:10.203276 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:10.203288 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:10.203370 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:10.203918 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:10.203930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.204707 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:10.204709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:10.205097 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:10.205103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:10.205257 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:10.206434 (+ 1177us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:10.206435 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:10.206463 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:10.206509 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:10.206789 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:10.206794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:10.206797 (+ 3us) client_negotiation.cc:769] Sending connection context
1210 14:59:10.206897 (+ 100us) client_negotiation.cc:240] Negotiation successful
1210 14:59:10.207048 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20251210 14:59:10.208563 16993 heartbeater.cc:344] Connected to a master server at 127.6.115.126:40073
I20251210 14:59:10.208734 16993 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:10.209162 16993 heartbeater.cc:507] Master 127.6.115.126:40073 requested a full tablet report, sending...
I20251210 14:59:10.210039 16543 ts_manager.cc:194] Registered new tserver with Master: f7a23f48d31e4a9da447a1bc1166327f (127.6.115.67:33029)
I20251210 14:59:10.210505 16543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.115.67:58773
I20251210 14:59:10.215538 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251210 14:59:10.217964 16582 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:10.217177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.6.115.126:40073)
1210 14:59:10.217304 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.217308 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.217479 (+ 171us) server_negotiation.cc:408] Connection header received
1210 14:59:10.217569 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.217572 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.217610 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.217678 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.217859 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55448: BlockingRecv error: recv got EOF from 127.0.0.1:55448 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20251210 14:59:10.218109 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16603
I20251210 14:59:10.224246 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16734
I20251210 14:59:10.229435 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16865
I20251210 14:59:10.234633 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 16512
2025-12-10T14:59:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (748 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-12-10T14:59:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:59:10Z Disabled control of system clock
I20251210 14:59:10.263530 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39625
--webserver_interface=127.6.115.126
--webserver_port=0
--builtin_ntp_servers=127.6.115.84:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251210 14:59:10.361657 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:10.361836 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:10.361879 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:10.364977 17004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:59:10.365027 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:10.365043 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:10.365061 17004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:59:10.365079 17004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:59:10.368736 17004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:39625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:10.369643 17004 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:10.370636 17004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:10.375656 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.375699 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.375674 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.375744 17004 server_base.cc:1047] running on GCE node
I20251210 14:59:10.376365 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:10.377138 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:10.378310 17004 hybrid_clock.cc:648] HybridClock initialized: now 1765378750378277 us; error 47 us; skew 500 ppm
I20251210 14:59:10.379927 17004 webserver.cc:492] Webserver started at http://127.6.115.126:33911/ using document root <none> and password file <none>
I20251210 14:59:10.380419 17004 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:10.380466 17004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:10.380606 17004 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:10.382110 17004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "fc0e02d451b543919d87dc6431fde280"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "5b16076978a5ac924ec94df01ab37f70"
server_key_iv: "4670b55524d739f1b322a2426fa205eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.382527 17004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "fc0e02d451b543919d87dc6431fde280"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "5b16076978a5ac924ec94df01ab37f70"
server_key_iv: "4670b55524d739f1b322a2426fa205eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.385521 17004 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251210 14:59:10.387534 17018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.388453 17004 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:10.388545 17004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "fc0e02d451b543919d87dc6431fde280"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "5b16076978a5ac924ec94df01ab37f70"
server_key_iv: "4670b55524d739f1b322a2426fa205eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.388636 17004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.392942 17004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.393445 17004 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.393579 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.399973 17070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:39625 every 8 connection(s)
I20251210 14:59:10.399981 17004 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:39625
I20251210 14:59:10.400887 17004 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:59:10.403422 17071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:10.407687 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17004
I20251210 14:59:10.407788 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:59:10.408000 6605 external_mini_cluster.cc:1468] Setting key 713c2d43528f86b864e367da3099555a
I20251210 14:59:10.408409 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280: Bootstrap starting.
I20251210 14:59:10.410535 17071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:10.411298 17071 log.cc:826] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:10.412678 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.409064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.6.115.126:39625)
1210 14:59:10.409613 (+ 549us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.409621 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.409653 (+ 32us) server_negotiation.cc:408] Connection header received
1210 14:59:10.410124 (+ 471us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.410150 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.410422 (+ 272us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.410703 (+ 281us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.411502 (+ 799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.411507 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.411515 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.411524 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.411561 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.411571 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.411666 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.411803 (+ 137us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.412010 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":109,"threads_started":1}
I20251210 14:59:10.413113 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280: No bootstrap required, opened a new log
I20251210 14:59:10.415581 17071 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } }
I20251210 14:59:10.415736 17071 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:10.415773 17071 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0e02d451b543919d87dc6431fde280, State: Initialized, Role: FOLLOWER
I20251210 14:59:10.416164 17071 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } }
I20251210 14:59:10.416273 17071 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:59:10.416333 17071 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:59:10.416417 17071 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:10.417127 17071 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } }
I20251210 14:59:10.417459 17071 leader_election.cc:304] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0e02d451b543919d87dc6431fde280; no voters:
I20251210 14:59:10.417697 17071 leader_election.cc:290] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:59:10.417807 17076 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:10.418022 17076 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [term 1 LEADER]: Becoming Leader. State: Replica: fc0e02d451b543919d87dc6431fde280, State: Running, Role: LEADER
I20251210 14:59:10.418320 17076 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } }
I20251210 14:59:10.418702 17071 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:59:10.419726 17078 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0e02d451b543919d87dc6431fde280. Latest consensus state: current_term: 1 leader_uuid: "fc0e02d451b543919d87dc6431fde280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } } }
I20251210 14:59:10.419842 17078 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:10.420188 17085 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:59:10.420737 17077 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0e02d451b543919d87dc6431fde280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0e02d451b543919d87dc6431fde280" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 39625 } } }
I20251210 14:59:10.420828 17077 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:10.422747 17085 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:59:10.427080 17085 catalog_manager.cc:1357] Generated new cluster ID: 555d819f31d747afb9d4b105054d7d6e
I20251210 14:59:10.427137 17085 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:59:10.435768 17085 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:59:10.436028 17085 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:59:10.458765 17085 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fc0e02d451b543919d87dc6431fde280: Generated new TSK 0
I20251210 14:59:10.459484 17085 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251210 14:59:10.462574 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--builtin_ntp_servers=127.6.115.84:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251210 14:59:10.567189 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:10.567373 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:10.567474 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:10.570463 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:10.570529 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:10.570629 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:59:10.574132 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:10.575191 17095 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:10.576210 17095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:10.581918 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.581946 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.582114 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.582211 17095 server_base.cc:1047] running on GCE node
I20251210 14:59:10.582502 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:10.582979 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:10.584127 17095 hybrid_clock.cc:648] HybridClock initialized: now 1765378750584117 us; error 30 us; skew 500 ppm
I20251210 14:59:10.585727 17095 webserver.cc:492] Webserver started at http://127.6.115.65:37815/ using document root <none> and password file <none>
I20251210 14:59:10.586229 17095 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:10.586282 17095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:10.586441 17095 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:10.587885 17095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "5cd1951de1d747e187f649b2e83da838"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "712b0248fb4e750f0baa6fae89c485ca"
server_key_iv: "4693770b16d7438616d611212351f166"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.588344 17095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "5cd1951de1d747e187f649b2e83da838"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "712b0248fb4e750f0baa6fae89c485ca"
server_key_iv: "4693770b16d7438616d611212351f166"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.591431 17095 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:59:10.593277 17109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.594218 17095 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:10.594318 17095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "5cd1951de1d747e187f649b2e83da838"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "712b0248fb4e750f0baa6fae89c485ca"
server_key_iv: "4693770b16d7438616d611212351f166"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.594401 17095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.598795 17095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.599272 17095 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.599411 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.599839 17095 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:10.600680 17095 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:10.600728 17095 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.600785 17095 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:10.600813 17095 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.608867 17095 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:39607
I20251210 14:59:10.608919 17222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:39607 every 8 connection(s)
I20251210 14:59:10.609736 17095 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:59:10.614179 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.611235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:54287 (local address 127.6.115.126:39625)
1210 14:59:10.611363 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.611366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.612147 (+ 781us) server_negotiation.cc:408] Connection header received
1210 14:59:10.612797 (+ 650us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.612800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.612837 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.612900 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.613844 (+ 944us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.613847 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.613849 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.613859 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.613879 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.613885 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.613959 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.614051 (+ 92us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.614082 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251210 14:59:10.614777 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.611455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39625 (local address 127.6.115.65:54287)
1210 14:59:10.612023 (+ 568us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:10.612052 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:10.612634 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:10.613017 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:10.613024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:10.613117 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:10.613649 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:10.613651 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:10.613676 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:10.613720 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:10.613951 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:10.613956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:10.613958 (+ 2us) client_negotiation.cc:769] Sending connection context
1210 14:59:10.614089 (+ 131us) client_negotiation.cc:240] Negotiation successful
1210 14:59:10.614262 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
I20251210 14:59:10.615860 17223 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39625
I20251210 14:59:10.616050 17223 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:10.616391 17223 heartbeater.cc:507] Master 127.6.115.126:39625 requested a full tablet report, sending...
I20251210 14:59:10.617223 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17095
I20251210 14:59:10.617353 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:59:10.617650 6605 external_mini_cluster.cc:1468] Setting key 5b012862d1645f2521804584a3eeafe0
I20251210 14:59:10.617767 17035 ts_manager.cc:194] Registered new tserver with Master: 5cd1951de1d747e187f649b2e83da838 (127.6.115.65:39607)
I20251210 14:59:10.619407 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--builtin_ntp_servers=127.6.115.84:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251210 14:59:10.718125 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:10.718307 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:10.718381 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:10.721539 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:10.721582 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:10.721678 17226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:59:10.724845 17226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:10.725795 17226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:10.726776 17226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:10.732554 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.732573 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.732599 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.733008 17226 server_base.cc:1047] running on GCE node
I20251210 14:59:10.733345 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:10.733837 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:10.734979 17226 hybrid_clock.cc:648] HybridClock initialized: now 1765378750734973 us; error 47 us; skew 500 ppm
I20251210 14:59:10.736510 17226 webserver.cc:492] Webserver started at http://127.6.115.66:36329/ using document root <none> and password file <none>
I20251210 14:59:10.736969 17226 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:10.737023 17226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:10.737174 17226 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:10.738698 17226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "0225db0f354a48cc90dcfb5f3a70914d"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "165779f66743739ba47163cb34f478e9"
server_key_iv: "ba95c1d404927a8fd258386e982e76d8"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.739144 17226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "0225db0f354a48cc90dcfb5f3a70914d"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "165779f66743739ba47163cb34f478e9"
server_key_iv: "ba95c1d404927a8fd258386e982e76d8"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.742122 17226 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251210 14:59:10.744011 17240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.744863 17226 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:10.744956 17226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "0225db0f354a48cc90dcfb5f3a70914d"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "165779f66743739ba47163cb34f478e9"
server_key_iv: "ba95c1d404927a8fd258386e982e76d8"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.745036 17226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.749337 17226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.749920 17226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.750064 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.750500 17226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:10.751266 17226 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:10.751309 17226 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.751368 17226 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:10.751394 17226 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.759976 17226 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:41497
I20251210 14:59:10.760007 17353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:41497 every 8 connection(s)
I20251210 14:59:10.760766 17226 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:59:10.764052 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17226
I20251210 14:59:10.764202 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:59:10.764470 6605 external_mini_cluster.cc:1468] Setting key 3c7d53dc4d6959b18e5b49e11ede52c3
I20251210 14:59:10.765592 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.762317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:42649 (local address 127.6.115.126:39625)
1210 14:59:10.762470 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.762474 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.763196 (+ 722us) server_negotiation.cc:408] Connection header received
1210 14:59:10.763902 (+ 706us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.763904 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.763941 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.763998 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.765046 (+ 1048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.765049 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.765052 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.765063 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.765105 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.765114 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.765213 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.765458 (+ 245us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.765491 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251210 14:59:10.766405 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.762543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39625 (local address 127.6.115.66:42649)
1210 14:59:10.763066 (+ 523us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:10.763096 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:10.763715 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:10.764161 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:10.764172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:10.764272 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:10.764849 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:10.764850 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:10.764877 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:10.764923 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:10.765239 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:10.765255 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:10.765260 (+ 5us) client_negotiation.cc:769] Sending connection context
1210 14:59:10.765526 (+ 266us) client_negotiation.cc:240] Negotiation successful
1210 14:59:10.765709 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20251210 14:59:10.766525 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--builtin_ntp_servers=127.6.115.84:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251210 14:59:10.767315 17354 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39625
I20251210 14:59:10.767534 17354 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:10.767927 17354 heartbeater.cc:507] Master 127.6.115.126:39625 requested a full tablet report, sending...
I20251210 14:59:10.768821 17034 ts_manager.cc:194] Registered new tserver with Master: 0225db0f354a48cc90dcfb5f3a70914d (127.6.115.66:41497)
W20251210 14:59:10.864027 17357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:10.864207 17357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:10.864289 17357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:10.867352 17357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:10.867395 17357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:10.867503 17357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:59:10.870988 17357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:10.871891 17357 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:10.872854 17357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:10.878379 17362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.878407 17363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:10.878432 17365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:10.878655 17357 server_base.cc:1047] running on GCE node
I20251210 14:59:10.878950 17357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:10.879395 17357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:10.880542 17357 hybrid_clock.cc:648] HybridClock initialized: now 1765378750880528 us; error 32 us; skew 500 ppm
I20251210 14:59:10.882092 17357 webserver.cc:492] Webserver started at http://127.6.115.67:39485/ using document root <none> and password file <none>
I20251210 14:59:10.882560 17357 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:10.882615 17357 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:10.882766 17357 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:10.884223 17357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "a0b554750cd44c9e8c4d3bf3b6f1b38b"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "7e8466e16412b1b69654ea920f9890b9"
server_key_iv: "ef1b176ea0d781147972df9308764e07"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.884635 17357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "a0b554750cd44c9e8c4d3bf3b6f1b38b"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "7e8466e16412b1b69654ea920f9890b9"
server_key_iv: "ef1b176ea0d781147972df9308764e07"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.887720 17357 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 14:59:10.889597 17371 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.890483 17357 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:10.890578 17357 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "a0b554750cd44c9e8c4d3bf3b6f1b38b"
format_stamp: "Formatted at 2025-12-10 14:59:10 on dist-test-slave-nb51"
server_key: "7e8466e16412b1b69654ea920f9890b9"
server_key_iv: "ef1b176ea0d781147972df9308764e07"
server_key_version: "encryptionkey@0"
I20251210 14:59:10.890686 17357 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:10.894825 17357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:10.895319 17357 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:10.895475 17357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:10.895893 17357 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:10.896663 17357 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:10.896704 17357 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.896737 17357 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:10.896759 17357 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:10.904963 17357 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:41185
I20251210 14:59:10.904989 17484 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:41185 every 8 connection(s)
I20251210 14:59:10.905797 17357 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:59:10.910565 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.907292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:51491 (local address 127.6.115.126:39625)
1210 14:59:10.907411 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.907414 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.908265 (+ 851us) server_negotiation.cc:408] Connection header received
1210 14:59:10.908976 (+ 711us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.908979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.909024 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.909122 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.910143 (+ 1021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:10.910146 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:10.910149 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1210 14:59:10.910160 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:10.910182 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1210 14:59:10.910190 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:10.910264 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:10.910395 (+ 131us) server_negotiation.cc:300] Negotiation successful
1210 14:59:10.910440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:59:10.911089 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17357
I20251210 14:59:10.911201 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:59:10.911192 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:10.907531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:39625 (local address 127.6.115.67:51491)
1210 14:59:10.908106 (+ 575us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:10.908161 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:10.908790 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:10.909280 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:10.909286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:10.909409 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:10.909941 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1210 14:59:10.909942 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:10.909968 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1210 14:59:10.910014 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:10.910259 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:10.910266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:10.910273 (+ 7us) client_negotiation.cc:769] Sending connection context
1210 14:59:10.910446 (+ 173us) client_negotiation.cc:240] Negotiation successful
1210 14:59:10.910629 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20251210 14:59:10.911504 6605 external_mini_cluster.cc:1468] Setting key 54ae4ccb4e389b9cbc7ec0b825b2ba93
I20251210 14:59:10.912281 17485 heartbeater.cc:344] Connected to a master server at 127.6.115.126:39625
I20251210 14:59:10.912518 17485 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:10.912950 17485 heartbeater.cc:507] Master 127.6.115.126:39625 requested a full tablet report, sending...
I20251210 14:59:10.913784 17034 ts_manager.cc:194] Registered new tserver with Master: a0b554750cd44c9e8c4d3bf3b6f1b38b (127.6.115.67:41185)
I20251210 14:59:10.924714 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251210 14:59:10.927193 17074 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 14:59:10.926396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.6.115.126:39625)
1210 14:59:10.926505 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:10.926509 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:10.926678 (+ 169us) server_negotiation.cc:408] Connection header received
1210 14:59:10.926795 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:10.926798 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:10.926832 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:10.926894 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:10.927107 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60338: BlockingRecv error: recv got EOF from 127.0.0.1:60338 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20251210 14:59:10.927753 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17095
I20251210 14:59:10.934255 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17226
I20251210 14:59:10.939456 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17357
I20251210 14:59:10.944617 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17004
2025-12-10T14:59:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (707 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:59:10 dist-test-slave-nb51 krb5kdc[17495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:59:10 dist-test-slave-nb51 krb5kdc[17495](info): set up 2 sockets
Dec 10 14:59:10 dist-test-slave-nb51 krb5kdc[17495](info): commencing operation
krb5kdc: starting...
W20251210 14:59:12.910028 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:59:12 dist-test-slave-nb51 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:59:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:59:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:13.041188 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:46841
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:13.139824 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:13.140050 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:13.140131 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:13.143160 17511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:59:13.143210 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:13.143244 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:13.143276 17511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:59:13.143311 17511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:59:13.147163 17511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:13.148008 17511 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:13.148728 17511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:13.153685 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.153692 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.153756 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.154073 17511 server_base.cc:1047] running on GCE node
I20251210 14:59:13.154675 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:13.155471 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:13.156634 17511 hybrid_clock.cc:648] HybridClock initialized: now 1765378753156621 us; error 32 us; skew 500 ppm
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:13.159188 17511 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:59:13.160207 17511 webserver.cc:492] Webserver started at http://127.6.115.126:46241/ using document root <none> and password file <none>
I20251210 14:59:13.160720 17511 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:13.160778 17511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:13.160956 17511 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:13.162573 17511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "c685512db0184f77b6365ea22ffdb597"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "4bbb94c923ee9540e06a994c3f3c9153"
server_key_iv: "ac24b4ba1ad2e8e781811f41c0c9165a"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.163026 17511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "c685512db0184f77b6365ea22ffdb597"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "4bbb94c923ee9540e06a994c3f3c9153"
server_key_iv: "ac24b4ba1ad2e8e781811f41c0c9165a"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.166503 17511 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:59:13.168474 17526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.169516 17511 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251210 14:59:13.169636 17511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "c685512db0184f77b6365ea22ffdb597"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "4bbb94c923ee9540e06a994c3f3c9153"
server_key_iv: "ac24b4ba1ad2e8e781811f41c0c9165a"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.169724 17511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:13.181499 17511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:13.186964 17511 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:13.187144 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:13.194265 17511 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:46841
I20251210 14:59:13.194269 17578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:46841 every 8 connection(s)
I20251210 14:59:13.195245 17511 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:59:13.195617 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17511
I20251210 14:59:13.195746 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:59:13.195991 6605 external_mini_cluster.cc:1468] Setting key 6191bee309c4bf6aca40b3661516bb79
I20251210 14:59:13.198491 17579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:13.204402 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: Bootstrap starting.
I20251210 14:59:13.206723 17579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:13.207515 17579 log.cc:826] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:13.208547 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.197106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.6.115.126:46841)
1210 14:59:13.197513 (+ 407us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.197521 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.197552 (+ 31us) server_negotiation.cc:408] Connection header received
1210 14:59:13.198136 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.198151 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.198414 (+ 263us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.198693 (+ 279us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:13.199775 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.200529 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.201475 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.201679 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.204297 (+ 2618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:13.204333 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:13.204342 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:13.204382 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:13.206094 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.206617 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.206621 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.206625 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.206691 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.206963 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.206966 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.206967 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.207293 (+ 326us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:13.207449 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.207766 (+ 317us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.207997 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
I20251210 14:59:13.209206 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: No bootstrap required, opened a new log
I20251210 14:59:13.211582 17579 raft_consensus.cc:359] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } }
I20251210 14:59:13.211777 17579 raft_consensus.cc:385] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:13.211849 17579 raft_consensus.cc:740] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685512db0184f77b6365ea22ffdb597, State: Initialized, Role: FOLLOWER
I20251210 14:59:13.212286 17579 consensus_queue.cc:260] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } }
I20251210 14:59:13.212421 17579 raft_consensus.cc:399] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:59:13.212491 17579 raft_consensus.cc:493] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:59:13.212585 17579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:13.213513 17579 raft_consensus.cc:515] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } }
I20251210 14:59:13.213802 17579 leader_election.cc:304] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c685512db0184f77b6365ea22ffdb597; no voters:
I20251210 14:59:13.214053 17579 leader_election.cc:290] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:59:13.214164 17584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:13.214354 17584 raft_consensus.cc:697] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [term 1 LEADER]: Becoming Leader. State: Replica: c685512db0184f77b6365ea22ffdb597, State: Running, Role: LEADER
I20251210 14:59:13.214620 17584 consensus_queue.cc:237] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } }
I20251210 14:59:13.215262 17579 sys_catalog.cc:565] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:59:13.216259 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c685512db0184f77b6365ea22ffdb597. Latest consensus state: current_term: 1 leader_uuid: "c685512db0184f77b6365ea22ffdb597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } } }
I20251210 14:59:13.216377 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:13.216657 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c685512db0184f77b6365ea22ffdb597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685512db0184f77b6365ea22ffdb597" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 46841 } } }
I20251210 14:59:13.216727 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:13.217267 17593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:59:13.219566 17593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:59:13.223946 17593 catalog_manager.cc:1357] Generated new cluster ID: c9429cda9cf64c1480eb3f49ea42287d
I20251210 14:59:13.223996 17593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:59:13.235174 17593 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:59:13.235904 17593 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:59:13.246011 17593 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: Generated new TSK 0
I20251210 14:59:13.246616 17593 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:13.290398 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:46841
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:13.388820 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:13.388993 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:13.389040 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:13.391992 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:13.392035 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:13.392095 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:59:13.395942 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:13.396770 17607 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:13.397449 17607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:13.403142 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.403178 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.403218 17607 server_base.cc:1047] running on GCE node
W20251210 14:59:13.403183 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.403680 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:13.404119 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:13.405262 17607 hybrid_clock.cc:648] HybridClock initialized: now 1765378753405242 us; error 42 us; skew 500 ppm
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:13.407552 17607 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:59:13.408411 17607 webserver.cc:492] Webserver started at http://127.6.115.65:46325/ using document root <none> and password file <none>
I20251210 14:59:13.408849 17607 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:13.408890 17607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:13.409032 17607 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:13.410519 17607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "1f0928191b014208a50b7d85ec5a3242"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "8c965e3a80138eee2d5c9ac054211948"
server_key_iv: "fcea59add9e5a808db23d56ba6ae19eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.410895 17607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "1f0928191b014208a50b7d85ec5a3242"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "8c965e3a80138eee2d5c9ac054211948"
server_key_iv: "fcea59add9e5a808db23d56ba6ae19eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.413789 17607 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:59:13.415732 17622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.416668 17607 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:13.416769 17607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "1f0928191b014208a50b7d85ec5a3242"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "8c965e3a80138eee2d5c9ac054211948"
server_key_iv: "fcea59add9e5a808db23d56ba6ae19eb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.416842 17607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:13.458235 17607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:13.461921 17607 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:13.462106 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:13.462688 17607 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:13.463653 17607 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:13.463711 17607 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.463757 17607 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:13.463784 17607 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.472337 17607 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:33509
I20251210 14:59:13.472373 17735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:33509 every 8 connection(s)
I20251210 14:59:13.473162 17607 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:59:13.475099 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17607
I20251210 14:59:13.475237 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:59:13.475472 6605 external_mini_cluster.cc:1468] Setting key a6bc7410aa39a4c40776b0ea7e0b3362
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 14:59:13.484916 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.474832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:41521 (local address 127.6.115.126:46841)
1210 14:59:13.474977 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.474982 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.475699 (+ 717us) server_negotiation.cc:408] Connection header received
1210 14:59:13.476652 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.476656 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.476701 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.476773 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:13.478411 (+ 1638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.478891 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.479500 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.479671 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.481694 (+ 2023us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:13.481708 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:13.481710 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:13.481732 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:13.483227 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.483653 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.483656 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.483658 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.483697 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.483952 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.483956 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.483958 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.484104 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:13.484198 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.484648 (+ 450us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.484764 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251210 14:59:13.485529 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.475131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:46841 (local address 127.6.115.65:41521)
1210 14:59:13.475565 (+ 434us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.475604 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.476431 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.476913 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.476922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.477277 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:13.478273 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.478287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.479002 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.479006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.479394 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.479401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.479588 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.480184 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:13.480206 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:13.481559 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:13.483339 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.483344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.483354 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.483559 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.483776 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.483781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.483783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.483868 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.484210 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:13.484218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:13.484410 (+ 192us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.484605 (+ 195us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.484834 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 14:59:13.486601 17736 heartbeater.cc:344] Connected to a master server at 127.6.115.126:46841
I20251210 14:59:13.486833 17736 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:13.487389 17736 heartbeater.cc:507] Master 127.6.115.126:46841 requested a full tablet report, sending...
I20251210 14:59:13.488694 17543 ts_manager.cc:194] Registered new tserver with Master: 1f0928191b014208a50b7d85ec5a3242 (127.6.115.65:33509)
I20251210 14:59:13.489792 17543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:41521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:13.517987 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:46841
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:13.615484 17743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:13.615665 17743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:13.615720 17743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:13.618825 17743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:13.618878 17743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:13.618952 17743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:59:13.622819 17743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:13.623653 17743 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:13.624397 17743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:13.630317 17749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.630348 17751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.630683 17743 server_base.cc:1047] running on GCE node
W20251210 14:59:13.630323 17748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.631042 17743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:13.631547 17743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:13.632714 17743 hybrid_clock.cc:648] HybridClock initialized: now 1765378753632679 us; error 39 us; skew 500 ppm
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:13.635063 17743 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:59:13.635949 17743 webserver.cc:492] Webserver started at http://127.6.115.66:44723/ using document root <none> and password file <none>
I20251210 14:59:13.636423 17743 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:13.636466 17743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:13.636597 17743 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:13.638125 17743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "d97cd98a340c4659b91313c3d0cacc63"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "35c332f9f0b20c9b44350dac8098fb68"
server_key_iv: "43021f1d7f8879c0a95c245a4b5200e8"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.638547 17743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "d97cd98a340c4659b91313c3d0cacc63"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "35c332f9f0b20c9b44350dac8098fb68"
server_key_iv: "43021f1d7f8879c0a95c245a4b5200e8"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.641592 17743 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 14:59:13.643545 17758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.644496 17743 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 14:59:13.644588 17743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "d97cd98a340c4659b91313c3d0cacc63"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "35c332f9f0b20c9b44350dac8098fb68"
server_key_iv: "43021f1d7f8879c0a95c245a4b5200e8"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.644670 17743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:13.678454 17743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:13.680868 17743 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:13.681033 17743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:13.681586 17743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:13.682344 17743 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:13.682387 17743 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.682430 17743 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:13.682461 17743 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.690922 17743 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:40523
I20251210 14:59:13.690966 17871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:40523 every 8 connection(s)
I20251210 14:59:13.691746 17743 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:59:13.692437 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17743
I20251210 14:59:13.692538 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:59:13.692776 6605 external_mini_cluster.cc:1468] Setting key 1fe918d3da9826b16e1f2786aab2d142
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
I20251210 14:59:13.704066 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.693345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:59229 (local address 127.6.115.126:46841)
1210 14:59:13.693485 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.693489 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.694248 (+ 759us) server_negotiation.cc:408] Connection header received
1210 14:59:13.695094 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.695100 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.695167 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.695258 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:13.696839 (+ 1581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.697665 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.698252 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.698482 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.700623 (+ 2141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:13.700644 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:13.700648 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:13.700677 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:13.702154 (+ 1477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.702630 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.702633 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.702634 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.702672 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.702887 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.702889 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.702890 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.703052 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:13.703157 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.703639 (+ 482us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.703781 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251210 14:59:13.704481 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.693664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:46841 (local address 127.6.115.66:59229)
1210 14:59:13.694124 (+ 460us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.694164 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.694898 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.695398 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.695404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.695742 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:13.696667 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.696680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.697770 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.697773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.698139 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.698146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.698301 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.698850 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:13.698872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:13.700471 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:13.702258 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.702263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.702271 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.702543 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.702736 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.702738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.702739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.702824 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.703167 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:13.703172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:13.703379 (+ 207us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.703539 (+ 160us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.703755 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20251210 14:59:13.705642 17872 heartbeater.cc:344] Connected to a master server at 127.6.115.126:46841
I20251210 14:59:13.705924 17872 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:13.706411 17872 heartbeater.cc:507] Master 127.6.115.126:46841 requested a full tablet report, sending...
I20251210 14:59:13.707470 17543 ts_manager.cc:194] Registered new tserver with Master: d97cd98a340c4659b91313c3d0cacc63 (127.6.115.66:40523)
I20251210 14:59:13.708290 17543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:59229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:13.736486 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:46841
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:13.834688 17879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:13.834877 17879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:13.834950 17879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:13.838038 17879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:13.838091 17879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:13.838165 17879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:59:13.842094 17879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:46841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:13.842921 17879 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:13.843631 17879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:13.849427 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.849423 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:13.849428 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:13.849764 17879 server_base.cc:1047] running on GCE node
I20251210 14:59:13.850078 17879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:13.850579 17879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:13.851732 17879 hybrid_clock.cc:648] HybridClock initialized: now 1765378753851718 us; error 28 us; skew 500 ppm
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:13.854249 17879 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:59:13.855237 17879 webserver.cc:492] Webserver started at http://127.6.115.67:44319/ using document root <none> and password file <none>
I20251210 14:59:13.855731 17879 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:13.855783 17879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:13.855943 17879 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251210 14:59:13.857478 17879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "ad4dcdc049984d4b8a8d4e47413cdc71"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "9877eb87f9c468a74ad859fc7399691b"
server_key_iv: "2ec11b8b8935ed40475ea411af5cd3bb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.857897 17879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "ad4dcdc049984d4b8a8d4e47413cdc71"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "9877eb87f9c468a74ad859fc7399691b"
server_key_iv: "2ec11b8b8935ed40475ea411af5cd3bb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.860848 17879 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251210 14:59:13.862793 17894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.863694 17879 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:59:13.863801 17879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "ad4dcdc049984d4b8a8d4e47413cdc71"
format_stamp: "Formatted at 2025-12-10 14:59:13 on dist-test-slave-nb51"
server_key: "9877eb87f9c468a74ad859fc7399691b"
server_key_iv: "2ec11b8b8935ed40475ea411af5cd3bb"
server_key_version: "encryptionkey@0"
I20251210 14:59:13.863888 17879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:13.893690 17879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:13.896263 17879 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:13.896435 17879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:13.896924 17879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:13.897722 17879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:13.897773 17879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.897825 17879 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:13.897858 17879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:13.906391 17879 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:43599
I20251210 14:59:13.906430 18007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:43599 every 8 connection(s)
I20251210 14:59:13.907277 17879 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:59:13.911422 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 17879
I20251210 14:59:13.911551 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:59:13.911787 6605 external_mini_cluster.cc:1468] Setting key b25dc1add3ee428d60f273d659b34331
Dec 10 14:59:13 dist-test-slave-nb51 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378753, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:13.919123 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.908892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:37539 (local address 127.6.115.126:46841)
1210 14:59:13.909057 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.909062 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.909726 (+ 664us) server_negotiation.cc:408] Connection header received
1210 14:59:13.910545 (+ 819us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.910548 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.910595 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.910665 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:13.912282 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.912938 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.913736 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.913916 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.915924 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:13.915938 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:13.915940 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:13.915964 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:13.917459 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.917879 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.917883 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.917885 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.917932 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.918203 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.918207 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.918209 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.918360 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:13.918447 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.918864 (+ 417us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.918977 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251210 14:59:13.919628 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.909126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:46841 (local address 127.6.115.67:37539)
1210 14:59:13.909568 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.909617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.910338 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.910788 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.910794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.911129 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:13.912128 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.912145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.913072 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.913076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.913628 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.913637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.913834 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.914395 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:13.914417 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:13.915783 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:13.917557 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.917562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.917571 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.917785 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.918010 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:13.918015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:13.918017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:13.918111 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:13.918457 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:13.918465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:13.918648 (+ 183us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.918814 (+ 166us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.919036 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20251210 14:59:13.920578 18008 heartbeater.cc:344] Connected to a master server at 127.6.115.126:46841
I20251210 14:59:13.920812 18008 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:13.921250 18008 heartbeater.cc:507] Master 127.6.115.126:46841 requested a full tablet report, sending...
I20251210 14:59:13.922094 17543 ts_manager.cc:194] Registered new tserver with Master: ad4dcdc049984d4b8a8d4e47413cdc71 (127.6.115.67:43599)
I20251210 14:59:13.922610 17543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:37539
I20251210 14:59:13.925063 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:59:13.933012 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.926788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.6.115.126:46841)
1210 14:59:13.926915 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.926919 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.927118 (+ 199us) server_negotiation.cc:408] Connection header received
1210 14:59:13.927284 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.927289 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.927334 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.927404 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:13.928298 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.928732 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.929356 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.929586 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.930522 (+ 936us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:13.930538 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:13.930540 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:13.930561 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:13.931647 (+ 1086us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.932023 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.932026 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.932028 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.932075 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:13.932288 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:13.932292 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:13.932293 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:13.932439 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:13.932554 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.932770 (+ 216us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.932878 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251210 14:59:13.935304 17543 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:59:13.937526 17543 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:59:13.949293 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.944815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43599 (local address 127.0.0.1:60958)
1210 14:59:13.945431 (+ 616us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.945469 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.945603 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.946357 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.946361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.946382 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:13.946704 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.946715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.948125 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.948128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.948923 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.948930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.949010 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.949022 (+ 12us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.949067 (+ 45us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.949149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"spinlock_wait_cycles":1792,"thread_start_us":67,"threads_started":1}
I20251210 14:59:13.949673 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.945495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:33509 (local address 127.0.0.1:54656)
1210 14:59:13.946303 (+ 808us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.946325 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.946423 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.946862 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.946865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.946882 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:13.947108 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.947118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.948402 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.948406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.949232 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.949268 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.949416 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.949433 (+ 17us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.949487 (+ 54us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.949553 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":79,"threads_started":1}
I20251210 14:59:13.950198 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.945528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.6.115.66:40523)
1210 14:59:13.945878 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.945884 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.946079 (+ 195us) server_negotiation.cc:408] Connection header received
1210 14:59:13.946306 (+ 227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.946314 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.946464 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.946575 (+ 111us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:13.947130 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.947829 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.949232 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.949782 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.949893 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.949964 (+ 71us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.950048 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20251210 14:59:13.950209 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.944998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40523 (local address 127.0.0.1:39244)
1210 14:59:13.945996 (+ 998us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:13.946009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:13.946138 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:13.946752 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:13.946758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:13.946779 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:13.947004 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.947014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.947934 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.947938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:13.948825 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:13.948834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.948936 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.948988 (+ 52us) client_negotiation.cc:769] Sending connection context
1210 14:59:13.949080 (+ 92us) client_negotiation.cc:240] Negotiation successful
1210 14:59:13.949149 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":103,"threads_started":1}
I20251210 14:59:13.950496 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.944912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.6.115.67:43599)
1210 14:59:13.945877 (+ 965us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.945884 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.945907 (+ 23us) server_negotiation.cc:408] Connection header received
1210 14:59:13.945999 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.946008 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.946188 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.946840 (+ 652us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:13.946977 (+ 137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.948017 (+ 1040us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.949548 (+ 1531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.950047 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.950143 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.950201 (+ 58us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.950365 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":106,"threads_started":1}
I20251210 14:59:13.950718 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:13.945761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.6.115.65:33509)
1210 14:59:13.946035 (+ 274us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:13.946044 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:13.946517 (+ 473us) server_negotiation.cc:408] Connection header received
1210 14:59:13.946619 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:13.946623 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:13.946738 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:13.946864 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:13.947245 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.948296 (+ 1051us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:13.949838 (+ 1542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:13.950355 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:13.950440 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:13.950501 (+ 61us) server_negotiation.cc:300] Negotiation successful
1210 14:59:13.950596 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20251210 14:59:13.953058 17806 tablet_service.cc:1505] Processing CreateTablet for tablet c4a50c9c98dc404aa97b8b881b0141be (DEFAULT_TABLE table=test-table [id=5c40122eaa6e4ad89c9a5d238b5c9233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:13.953056 17670 tablet_service.cc:1505] Processing CreateTablet for tablet c4a50c9c98dc404aa97b8b881b0141be (DEFAULT_TABLE table=test-table [id=5c40122eaa6e4ad89c9a5d238b5c9233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:13.953845 17670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a50c9c98dc404aa97b8b881b0141be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:13.953856 17806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a50c9c98dc404aa97b8b881b0141be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:13.958791 18026 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Bootstrap starting.
I20251210 14:59:13.959231 18025 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Bootstrap starting.
I20251210 14:59:13.960505 17942 tablet_service.cc:1505] Processing CreateTablet for tablet c4a50c9c98dc404aa97b8b881b0141be (DEFAULT_TABLE table=test-table [id=5c40122eaa6e4ad89c9a5d238b5c9233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:13.960870 18026 tablet_bootstrap.cc:654] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:13.961068 18025 tablet_bootstrap.cc:654] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:13.961334 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a50c9c98dc404aa97b8b881b0141be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:13.961731 18026 log.cc:826] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:13.961731 18025 log.cc:826] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:13.963407 18026 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: No bootstrap required, opened a new log
I20251210 14:59:13.963407 18025 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: No bootstrap required, opened a new log
I20251210 14:59:13.963582 18026 ts_tablet_manager.cc:1403] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251210 14:59:13.963582 18025 ts_tablet_manager.cc:1403] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251210 14:59:13.965637 18029 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Bootstrap starting.
I20251210 14:59:13.966897 18025 raft_consensus.cc:359] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.967113 18025 raft_consensus.cc:385] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:13.967172 18025 raft_consensus.cc:740] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0928191b014208a50b7d85ec5a3242, State: Initialized, Role: FOLLOWER
I20251210 14:59:13.967048 18026 raft_consensus.cc:359] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.967248 18026 raft_consensus.cc:385] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:13.967303 18026 raft_consensus.cc:740] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97cd98a340c4659b91313c3d0cacc63, State: Initialized, Role: FOLLOWER
I20251210 14:59:13.967597 18025 consensus_queue.cc:260] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.967638 18026 consensus_queue.cc:260] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.968426 18029 tablet_bootstrap.cc:654] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:13.968432 18026 ts_tablet_manager.cc:1434] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20251210 14:59:13.968513 18025 ts_tablet_manager.cc:1434] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251210 14:59:13.968801 17736 heartbeater.cc:499] Master 127.6.115.126:46841 was elected leader, sending a full tablet report...
I20251210 14:59:13.968799 17872 heartbeater.cc:499] Master 127.6.115.126:46841 was elected leader, sending a full tablet report...
I20251210 14:59:13.969333 18029 log.cc:826] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:13.970839 18029 tablet_bootstrap.cc:492] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: No bootstrap required, opened a new log
I20251210 14:59:13.971035 18029 ts_tablet_manager.cc:1403] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251210 14:59:13.973976 18029 raft_consensus.cc:359] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.974195 18029 raft_consensus.cc:385] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251210 14:59:13.974174 17737 tablet.cc:2378] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:59:13.974256 18029 raft_consensus.cc:740] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4dcdc049984d4b8a8d4e47413cdc71, State: Initialized, Role: FOLLOWER
I20251210 14:59:13.974634 18029 consensus_queue.cc:260] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:13.975171 18008 heartbeater.cc:499] Master 127.6.115.126:46841 was elected leader, sending a full tablet report...
I20251210 14:59:13.975332 18029 ts_tablet_manager.cc:1434] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20251210 14:59:14.158094 18009 tablet.cc:2378] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251210 14:59:14.192711 17873 tablet.cc:2378] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:59:14.229184 18031 raft_consensus.cc:493] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:59:14.229390 18031 raft_consensus.cc:515] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:14.230664 18031 leader_election.cc:290] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad4dcdc049984d4b8a8d4e47413cdc71 (127.6.115.67:43599), d97cd98a340c4659b91313c3d0cacc63 (127.6.115.66:40523)
I20251210 14:59:14.234262 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.230829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:40523 (local address 127.6.115.65:47939)
1210 14:59:14.231150 (+ 321us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:14.231166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:14.231289 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:14.231573 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:14.231576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:14.231614 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:14.231894 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:14.231901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.232816 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:14.232820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:14.233807 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:14.233818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.233940 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:14.233961 (+ 21us) client_negotiation.cc:769] Sending connection context
1210 14:59:14.234017 (+ 56us) client_negotiation.cc:240] Negotiation successful
1210 14:59:14.234078 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"spinlock_wait_cycles":6912,"thread_start_us":76,"threads_started":1}
I20251210 14:59:14.234262 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.230829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:43599 (local address 127.6.115.65:58901)
1210 14:59:14.231242 (+ 413us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:14.231260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:14.231328 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:14.231573 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:14.231576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:14.231613 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:14.231896 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:14.231901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.232816 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:14.232820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:14.233807 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:14.233818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.233939 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:14.233961 (+ 22us) client_negotiation.cc:769] Sending connection context
1210 14:59:14.234027 (+ 66us) client_negotiation.cc:240] Negotiation successful
1210 14:59:14.234077 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":51,"spinlock_wait_cycles":13824,"thread_start_us":121,"threads_started":1}
I20251210 14:59:14.234661 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.230973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:58901 (local address 127.6.115.67:43599)
1210 14:59:14.231168 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:14.231173 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:14.231315 (+ 142us) server_negotiation.cc:408] Connection header received
1210 14:59:14.231419 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:14.231422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:14.231469 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:14.231554 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:14.232032 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.232706 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:14.233938 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.234424 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:14.234459 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:14.234507 (+ 48us) server_negotiation.cc:300] Negotiation successful
1210 14:59:14.234556 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20251210 14:59:14.234661 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.231023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:47939 (local address 127.6.115.66:40523)
1210 14:59:14.231169 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:14.231174 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:14.231253 (+ 79us) server_negotiation.cc:408] Connection header received
1210 14:59:14.231417 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:14.231420 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:14.231469 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:14.231554 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:14.232014 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.232706 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:14.233938 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.234427 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:14.234459 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:14.234507 (+ 48us) server_negotiation.cc:300] Negotiation successful
1210 14:59:14.234557 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251210 14:59:14.235169 17962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4a50c9c98dc404aa97b8b881b0141be" candidate_uuid: "1f0928191b014208a50b7d85ec5a3242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" is_pre_election: true
I20251210 14:59:14.235193 17826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4a50c9c98dc404aa97b8b881b0141be" candidate_uuid: "1f0928191b014208a50b7d85ec5a3242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97cd98a340c4659b91313c3d0cacc63" is_pre_election: true
I20251210 14:59:14.235406 17962 raft_consensus.cc:2468] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0928191b014208a50b7d85ec5a3242 in term 0.
I20251210 14:59:14.235424 17826 raft_consensus.cc:2468] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0928191b014208a50b7d85ec5a3242 in term 0.
I20251210 14:59:14.235796 17625 leader_election.cc:304] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0928191b014208a50b7d85ec5a3242, ad4dcdc049984d4b8a8d4e47413cdc71; no voters:
I20251210 14:59:14.236006 18031 raft_consensus.cc:2804] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:59:14.236075 18031 raft_consensus.cc:493] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:59:14.236105 18031 raft_consensus.cc:3060] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:14.237010 18031 raft_consensus.cc:515] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:14.237296 18031 leader_election.cc:290] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [CANDIDATE]: Term 1 election: Requested vote from peers ad4dcdc049984d4b8a8d4e47413cdc71 (127.6.115.67:43599), d97cd98a340c4659b91313c3d0cacc63 (127.6.115.66:40523)
I20251210 14:59:14.237623 17962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4a50c9c98dc404aa97b8b881b0141be" candidate_uuid: "1f0928191b014208a50b7d85ec5a3242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71"
I20251210 14:59:14.237645 17826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4a50c9c98dc404aa97b8b881b0141be" candidate_uuid: "1f0928191b014208a50b7d85ec5a3242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97cd98a340c4659b91313c3d0cacc63"
I20251210 14:59:14.237737 17962 raft_consensus.cc:3060] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:14.237737 17826 raft_consensus.cc:3060] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:14.238829 17962 raft_consensus.cc:2468] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0928191b014208a50b7d85ec5a3242 in term 1.
I20251210 14:59:14.238829 17826 raft_consensus.cc:2468] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0928191b014208a50b7d85ec5a3242 in term 1.
I20251210 14:59:14.239120 17626 leader_election.cc:304] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0928191b014208a50b7d85ec5a3242, d97cd98a340c4659b91313c3d0cacc63; no voters:
I20251210 14:59:14.239291 18031 raft_consensus.cc:2804] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:14.239496 18031 raft_consensus.cc:697] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0928191b014208a50b7d85ec5a3242, State: Running, Role: LEADER
I20251210 14:59:14.239777 18031 consensus_queue.cc:237] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } }
I20251210 14:59:14.242620 17543 catalog_manager.cc:5654] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0928191b014208a50b7d85ec5a3242 (127.6.115.65). New cstate: current_term: 1 leader_uuid: "1f0928191b014208a50b7d85ec5a3242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0928191b014208a50b7d85ec5a3242" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 33509 } health_report { overall_health: HEALTHY } } }
I20251210 14:59:14.308166 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.305061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.6.115.65:33509)
1210 14:59:14.305232 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:14.305236 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:14.305248 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:59:14.305407 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:14.305409 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:14.305459 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:14.305536 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:59:14.305879 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.306372 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:14.307095 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:14.307259 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:14.307358 (+ 99us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:59:14.307750 (+ 392us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:59:14.307848 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:14.307986 (+ 138us) server_negotiation.cc:300] Negotiation successful
1210 14:59:14.308031 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251210 14:59:14.313273 17962 raft_consensus.cc:1275] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 1 FOLLOWER]: Refusing update from remote peer 1f0928191b014208a50b7d85ec5a3242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:59:14.313274 17826 raft_consensus.cc:1275] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 1 FOLLOWER]: Refusing update from remote peer 1f0928191b014208a50b7d85ec5a3242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:59:14.313874 18031 consensus_queue.cc:1048] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4dcdc049984d4b8a8d4e47413cdc71" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 43599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:59:14.314011 18036 consensus_queue.cc:1048] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97cd98a340c4659b91313c3d0cacc63" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:59:14.321049 18040 mvcc.cc:204] Tried to move back new op lower bound from 7230991377660133376 to 7230991377368379392. Current Snapshot: MvccSnapshot[applied={T|T < 7230991377660133376}]
I20251210 14:59:14.327808 17543 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:59:14.327932 17543 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:59:14.330016 17543 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c685512db0184f77b6365ea22ffdb597: Sending DeleteTablet for 3 replicas of tablet c4a50c9c98dc404aa97b8b881b0141be
I20251210 14:59:14.330693 17942 tablet_service.cc:1552] Processing DeleteTablet for tablet c4a50c9c98dc404aa97b8b881b0141be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:14 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:60958
I20251210 14:59:14.330825 17806 tablet_service.cc:1552] Processing DeleteTablet for tablet c4a50c9c98dc404aa97b8b881b0141be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:14 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:39244
I20251210 14:59:14.330833 17670 tablet_service.cc:1552] Processing DeleteTablet for tablet c4a50c9c98dc404aa97b8b881b0141be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:14 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:54656
I20251210 14:59:14.331265 18049 tablet_replica.cc:333] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242: stopping tablet replica
I20251210 14:59:14.331444 18050 tablet_replica.cc:333] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: stopping tablet replica
I20251210 14:59:14.331532 18049 raft_consensus.cc:2243] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 1 LEADER]: Raft consensus shutting down.
I20251210 14:59:14.331727 18050 raft_consensus.cc:2243] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:59:14.331843 18049 raft_consensus.cc:2272] T c4a50c9c98dc404aa97b8b881b0141be P 1f0928191b014208a50b7d85ec5a3242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:59:14.331939 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17607
I20251210 14:59:14.332115 18050 raft_consensus.cc:2272] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:59:14.333298 18050 ts_tablet_manager.cc:1916] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:59:14.335294 18050 ts_tablet_manager.cc:1929] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:59:14.335373 18050 log.cc:1199] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/c4a50c9c98dc404aa97b8b881b0141be
I20251210 14:59:14.335660 18050 ts_tablet_manager.cc:1950] T c4a50c9c98dc404aa97b8b881b0141be P d97cd98a340c4659b91313c3d0cacc63: Deleting consensus metadata
I20251210 14:59:14.336560 17530 catalog_manager.cc:4985] TS d97cd98a340c4659b91313c3d0cacc63 (127.6.115.66:40523): tablet c4a50c9c98dc404aa97b8b881b0141be (table test-table [id=5c40122eaa6e4ad89c9a5d238b5c9233]) successfully deleted
W20251210 14:59:14.338845 17529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:33509 (error 108)
I20251210 14:59:14.339003 18048 tablet_replica.cc:333] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: stopping tablet replica
I20251210 14:59:14.339257 18048 raft_consensus.cc:2243] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:59:14.339548 18048 raft_consensus.cc:2272] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:59:14.339591 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17743
I20251210 14:59:14.339776 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:14.339487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:33509 (local address 127.0.0.1:54668)
1210 14:59:14.339629 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:14.339690 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:33509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20251210 14:59:14.341486 18048 ts_tablet_manager.cc:1916] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251210 14:59:14.341804 17529 catalog_manager.cc:4712] TS 1f0928191b014208a50b7d85ec5a3242 (127.6.115.65:33509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4a50c9c98dc404aa97b8b881b0141be: Network error: Client connection negotiation failed: client connection to 127.6.115.65:33509: connect: Connection refused (error 111)
I20251210 14:59:14.344527 18048 ts_tablet_manager.cc:1929] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:59:14.344616 18048 log.cc:1199] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/c4a50c9c98dc404aa97b8b881b0141be
I20251210 14:59:14.344960 18048 ts_tablet_manager.cc:1950] T c4a50c9c98dc404aa97b8b881b0141be P ad4dcdc049984d4b8a8d4e47413cdc71: Deleting consensus metadata
I20251210 14:59:14.346102 17529 catalog_manager.cc:4985] TS ad4dcdc049984d4b8a8d4e47413cdc71 (127.6.115.67:43599): tablet c4a50c9c98dc404aa97b8b881b0141be (table test-table [id=5c40122eaa6e4ad89c9a5d238b5c9233]) successfully deleted
I20251210 14:59:14.346625 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17879
I20251210 14:59:14.352370 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 17511
2025-12-10T14:59:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3409 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 14:59:14 dist-test-slave-nb51 krb5kdc[18054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 14:59:14 dist-test-slave-nb51 krb5kdc[18054](info): set up 2 sockets
Dec 10 14:59:14 dist-test-slave-nb51 krb5kdc[18054](info): commencing operation
krb5kdc: starting...
W20251210 14:59:16.294013 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.920s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 14:59:16 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T14:59:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T14:59:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangeradmin_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangeradmin_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangerlookup_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangerlookup_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
I20251210 14:59:16.435838 6605 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskceAy_x/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/postgres -l logfile start
2025-12-10 14:59:18.751 UTC [18085] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-10 14:59:18.751 UTC [18085] LOG: listening on IPv4 address "127.6.115.84", port 56929
2025-12-10 14:59:18.754 UTC [18085] LOG: listening on Unix socket "/tmp/.s.PGSQL.56929"
2025-12-10 14:59:18.758 UTC [18090] LOG: database system was shut down at 2025-12-10 14:59:17 UTC
2025-12-10 14:59:18.762 UTC [18085] LOG: database system is ready to accept connections
I20251210 14:59:20.621697 6605 mini_postgres.cc:96] Postgres bound to 56929
2025-12-10 14:59:20.627 UTC [18096] FATAL: database "slave" does not exist
127.6.115.84:56929 - accepting connections
I20251210 14:59:20.628293 6605 mini_ranger.cc:161] Starting Ranger...
I20251210 14:59:20.645396 6605 mini_ranger.cc:84] Created miniranger Postgres user
I20251210 14:59:20.710918 6605 mini_ranger.cc:87] Created ranger Postgres database
I20251210 14:59:20.711007 6605 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin
2025-12-10 14:59:21,000 [I] DB FLAVOR :POSTGRES
2025-12-10 14:59:21,001 [I] --------- Verifying Ranger DB connection ---------
2025-12-10 14:59:21,001 [I] Checking connection..
2025-12-10 14:59:21,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-10 14:59:21,317 [I] Checking connection passed.
2025-12-10 14:59:21,317 [I] --------- Verifying version history table ---------
2025-12-10 14:59:21,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 14:59:21,612 [I] Table x_db_version_h does not exist in database ranger
2025-12-10 14:59:21,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 14:59:21,927 [I] Table x_db_version_h does not exist in database ranger
2025-12-10 14:59:21,928 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-10 14:59:21,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-10 14:59:22.231 UTC [18163] WARNING: there is no transaction in progress
2025-12-10 14:59:22,237 [I] create_dbversion_catalog.sql file imported successfully
2025-12-10 14:59:22,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 14:59:22,542 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-10 14:59:22,542 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-10 14:59:22,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-10 14:59:22,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-10 14:59:23,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nb51.c.gcp-upstream.internal','N') ;"
2025-12-10 14:59:23,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-10 14:59:23,724 [I] Table x_portal_user does not exist in database ranger
2025-12-10 14:59:23,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-10 14:59:24,014 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-10 14:59:24,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-10 14:59:24,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-10 14:59:24,591 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-10 14:59:24,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-10 14:59:25.356 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.367 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.378 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.388 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.399 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.411 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.467 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.471 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.481 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.492 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.503 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.512 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.523 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.527 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.530 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25.533 UTC [18280] WARNING: there is no transaction in progress
2025-12-10 14:59:25,921 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-10 14:59:25,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-10 14:59:26,211 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-10 14:59:26,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-10 14:59:26,508 [I] Table x_portal_user already exists in database 'ranger'
2025-12-10 14:59:26,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-10 14:59:26,794 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-10 14:59:26,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-10 14:59:27,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-10 14:59:27,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nb51.c.gcp-upstream.internal';"
2025-12-10 14:59:27,672 [I] CORE_DB_SCHEMA import status has been updated
2025-12-10 14:59:27,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-10 14:59:27,959 [I] DB_PATCHES have already been applied
I20251210 14:59:27.964740 6605 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskceAy_x/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251210 14:59:27.964802 6605 mini_ranger.cc:193] Using host: 127.6.115.84
I20251210 14:59:27.967610 6605 mini_ranger.cc:227] Ranger admin URL: http://127.6.115.84:58543
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp].
Dec 10, 2025 2:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20251210 14:59:30.140475 6605 mini_ranger.cc:160] Time spent starting Ranger: real 9.512s user 0.002s sys 0.007s
Dec 10, 2025 2:59:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 10, 2025 2:59:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
Dec 10, 2025 2:59:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 10, 2025 2:59:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 10, 2025 2:59:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20251210 14:59:49.020941 6605 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangerkms_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/rangerkms_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.115.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.115.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 10 14:59:49 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251210 14:59:49.090894 6605 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251210 14:59:49.099506 6605 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251210 14:59:49.201898 6605 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251210 14:59:49.201993 6605 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms
I20251210 14:59:49.202030 6605 mini_ranger_kms.cc:227] Using postgres at 127.6.115.84:56929
2025-12-10 14:59:49,393 [I] DB FLAVOR :POSTGRES
2025-12-10 14:59:49,393 [I] --------- Verifying Ranger DB connection ---------
2025-12-10 14:59:49,393 [I] Checking connection
2025-12-10 14:59:49,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-10 14:59:49,726 [I] connection success
2025-12-10 14:59:49,726 [I] --------- Verifying Ranger DB tables ---------
2025-12-10 14:59:49,726 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-10 14:59:49,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-10 14:59:49,997 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-10 14:59:49,998 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-10 14:59:49,998 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-10 14:59:49,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:56929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-10 14:59:50,281 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251210 14:59:50.723691 6605 mini_ranger_kms.cc:310] Created kms service
I20251210 14:59:50.917652 6605 mini_ranger_kms.cc:326] Created kudu user
I20251210 14:59:50.957062 6605 mini_ranger_kms.cc:343] Created rangerkms user
Dec 10 14:59:50 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:51.319893 6605 mini_ranger_kms.cc:384] Added ranger policy
I20251210 14:59:51.320083 6605 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms:/tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251210 14:59:51.320119 6605 mini_ranger_kms.cc:240] Using host: 127.6.115.84
I20251210 14:59:51.323598 6605 mini_ranger_kms.cc:276] Ranger KMS PID: 18472
I20251210 14:59:51.323673 6605 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.115.84:48861
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp].
Dec 10, 2025 2:59:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20251210 14:59:53.557766 6605 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.467s user 0.013s sys 0.161s
I20251210 14:59:53.557912 6605 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251210 14:59:53.558018 6605 mini_ranger_kms.cc:401] 127.6.115.84:48861/kms/v1/keys
Dec 10 14:59:53 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765378793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Dec 10 14:59:54 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
Dec 10 14:59:54 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
Dec 10 14:59:56 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:56.488312 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:44389
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:44389
--ranger_config_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:48861/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:56.608289 18525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:56.608484 18525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:56.608562 18525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:56.611693 18525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 14:59:56.611752 18525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:56.611786 18525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:56.611820 18525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 14:59:56.611860 18525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 14:59:56.616012 18525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.115.84:48861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:44389
--ranger_config_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:56.616917 18525 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:56.617676 18525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:56.622821 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:56.622825 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:56.622864 18525 server_base.cc:1047] running on GCE node
W20251210 14:59:56.622874 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:56.623518 18525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:56.624289 18525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:56.625423 18525 hybrid_clock.cc:648] HybridClock initialized: now 1765378796625408 us; error 35 us; skew 500 ppm
Dec 10 14:59:56 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378796, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:56.627980 18525 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 14:59:56.629020 18525 webserver.cc:492] Webserver started at http://127.6.115.126:44237/ using document root <none> and password file <none>
I20251210 14:59:56.629580 18525 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:56.629633 18525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:56.629810 18525 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 14:59:56 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378796, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:56.682061 18525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "1e44d855812d4f86905718fa46550eb4"
format_stamp: "Formatted at 2025-12-10 14:59:56 on dist-test-slave-nb51"
server_key: "71e09f340d26b2ed38bdb8440c5d439a"
server_key_iv: "da61d2c939e56819055b0a9bafdecabd"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:56.682574 18525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "1e44d855812d4f86905718fa46550eb4"
format_stamp: "Formatted at 2025-12-10 14:59:56 on dist-test-slave-nb51"
server_key: "71e09f340d26b2ed38bdb8440c5d439a"
server_key_iv: "da61d2c939e56819055b0a9bafdecabd"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:56.685962 18525 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251210 14:59:56.702697 18540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:56.703760 18525 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251210 14:59:56.703872 18525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "1e44d855812d4f86905718fa46550eb4"
format_stamp: "Formatted at 2025-12-10 14:59:56 on dist-test-slave-nb51"
server_key: "71e09f340d26b2ed38bdb8440c5d439a"
server_key_iv: "da61d2c939e56819055b0a9bafdecabd"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:56.703996 18525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:56.717540 18525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:56.720288 18525 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:56.720467 18525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:56.728700 18525 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:44389
I20251210 14:59:56.728731 18592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:44389 every 8 connection(s)
I20251210 14:59:56.729720 18525 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 14:59:56.732338 18593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:56.734578 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 18525
I20251210 14:59:56.734709 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 14:59:56.734999 6605 external_mini_cluster.cc:1468] Setting key 5bcab51e270c98c71297926e267769b0
I20251210 14:59:56.737890 18593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: Bootstrap starting.
I20251210 14:59:56.740481 18593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:56.741228 18593 log.cc:826] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:56.743196 18593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: No bootstrap required, opened a new log
Dec 10 14:59:56 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:56.746451 18593 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } }
I20251210 14:59:56.746657 18593 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:56.746766 18593 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e44d855812d4f86905718fa46550eb4, State: Initialized, Role: FOLLOWER
I20251210 14:59:56.747225 18593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } }
I20251210 14:59:56.747355 18593 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 14:59:56.747437 18593 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 14:59:56.747543 18593 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:56.748273 18593 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } }
I20251210 14:59:56.748654 18593 leader_election.cc:304] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e44d855812d4f86905718fa46550eb4; no voters:
I20251210 14:59:56.748976 18593 leader_election.cc:290] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 14:59:56.749073 18598 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:56.749910 18598 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1e44d855812d4f86905718fa46550eb4, State: Running, Role: LEADER
I20251210 14:59:56.750224 18598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } }
I20251210 14:59:56.750489 18593 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20251210 14:59:56.750663 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:56.736269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.6.115.126:44389)
1210 14:59:56.736723 (+ 454us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:56.736736 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:56.736778 (+ 42us) server_negotiation.cc:408] Connection header received
1210 14:59:56.737463 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:56.737488 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:56.737835 (+ 347us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:56.738188 (+ 353us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:56.739248 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:56.740373 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:56.741257 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:56.741620 (+ 363us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:56.744120 (+ 2500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:56.744138 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:56.744151 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:56.744190 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:56.746986 (+ 2796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:56.747415 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:56.747420 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:56.747425 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:56.747506 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:56.748663 (+ 1157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:56.748668 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:56.748670 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:56.749068 (+ 398us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:56.749277 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:56.749604 (+ 327us) server_negotiation.cc:300] Negotiation successful
1210 14:59:56.749818 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20251210 14:59:56.751384 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e44d855812d4f86905718fa46550eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } } }
I20251210 14:59:56.751543 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:56.751784 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e44d855812d4f86905718fa46550eb4. Latest consensus state: current_term: 1 leader_uuid: "1e44d855812d4f86905718fa46550eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e44d855812d4f86905718fa46550eb4" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44389 } } }
I20251210 14:59:56.751888 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4 [sys.catalog]: This master's current role is: LEADER
I20251210 14:59:56.753266 18593 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251210 14:59:56.758402 18607 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 14:59:56.761804 18607 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 14:59:56.768023 18607 catalog_manager.cc:1357] Generated new cluster ID: 7296a5d5c5504b4e9f91ed8aae519180
I20251210 14:59:56.768081 18607 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 14:59:56.786283 18607 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 14:59:56.787344 18607 catalog_manager.cc:1514] Loading token signing keys...
I20251210 14:59:56.793169 18607 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: Generated new TSK 0
I20251210 14:59:56.793893 18607 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:57.494132 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:48861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:44389
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:57.606235 18639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:57.606402 18639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:57.606452 18639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:57.609772 18639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:57.609828 18639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:57.609920 18639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 14:59:57.613818 18639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.115.84:48861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:57.614702 18639 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:57.615461 18639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:57.621310 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:57.621311 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:57.621649 18639 server_base.cc:1047] running on GCE node
W20251210 14:59:57.621879 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:57.622188 18639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:57.622650 18639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:57.623814 18639 hybrid_clock.cc:648] HybridClock initialized: now 1765378797623711 us; error 125 us; skew 500 ppm
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:57.626199 18639 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 14:59:57.627166 18639 webserver.cc:492] Webserver started at http://127.6.115.65:43427/ using document root <none> and password file <none>
I20251210 14:59:57.627651 18639 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:57.627714 18639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:57.627894 18639 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:57.642091 18639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "59f6512c9bcb407eb10d1d2abe7b22ce"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "f8fc4e41a33b9c09dbc1e51feaec5b7e"
server_key_iv: "4ea08cb10961ed63e0af60b4af244f1e"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.642722 18639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "59f6512c9bcb407eb10d1d2abe7b22ce"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "f8fc4e41a33b9c09dbc1e51feaec5b7e"
server_key_iv: "4ea08cb10961ed63e0af60b4af244f1e"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.662740 18639 fs_manager.cc:696] Time spent creating directory manager: real 0.020s user 0.003s sys 0.000s
I20251210 14:59:57.681528 18657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:57.682631 18639 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251210 14:59:57.682740 18639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "59f6512c9bcb407eb10d1d2abe7b22ce"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "f8fc4e41a33b9c09dbc1e51feaec5b7e"
server_key_iv: "4ea08cb10961ed63e0af60b4af244f1e"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.682855 18639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:57.708734 18639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:57.712373 18639 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:57.712631 18639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:57.713335 18639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:57.714497 18639 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:57.714614 18639 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:57.714735 18639 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:57.714835 18639 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:57.729516 18639 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:35789
I20251210 14:59:57.730840 18639 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 14:59:57.731678 18771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:35789 every 8 connection(s)
I20251210 14:59:57.740382 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 18639
I20251210 14:59:57.740505 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 14:59:57.740818 6605 external_mini_cluster.cc:1468] Setting key d2d6646b8911b623f1ebcf35c0c67154
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:57.751418 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:57.734033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44389 (local address 127.6.115.65:43665)
1210 14:59:57.735658 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:57.735707 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:57.736613 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:57.737544 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:57.737554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:57.737876 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:57.738576 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:57.738590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:57.739726 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:57.739730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:57.741494 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:57.741503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:57.741714 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:57.744739 (+ 3025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:57.744798 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:57.746432 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:57.748620 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:57.748626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:57.748637 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:57.748949 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:57.749227 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:57.749230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:57.749232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:57.749344 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:57.749835 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:57.749843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:57.750116 (+ 273us) client_negotiation.cc:769] Sending connection context
1210 14:59:57.750341 (+ 225us) client_negotiation.cc:240] Negotiation successful
1210 14:59:57.750591 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"thread_start_us":100,"threads_started":1}
I20251210 14:59:57.752125 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:57.735034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:43665 (local address 127.6.115.126:44389)
1210 14:59:57.737049 (+ 2015us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:57.737053 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:57.737067 (+ 14us) server_negotiation.cc:408] Connection header received
1210 14:59:57.737107 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:57.737110 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:57.737164 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:57.737262 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:57.738732 (+ 1470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:57.739576 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:57.742260 (+ 2684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:57.744314 (+ 2054us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:57.746608 (+ 2294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:57.746628 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:57.746631 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:57.746654 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:57.748473 (+ 1819us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:57.749066 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:57.749070 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:57.749072 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:57.749122 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:57.749476 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:57.749479 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:57.749481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:57.749671 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:57.749776 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:57.751829 (+ 2053us) server_negotiation.cc:300] Negotiation successful
1210 14:59:57.751972 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1928,"thread_start_us":108,"threads_started":1}
I20251210 14:59:57.753726 18772 heartbeater.cc:344] Connected to a master server at 127.6.115.126:44389
I20251210 14:59:57.754020 18772 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:57.754602 18772 heartbeater.cc:507] Master 127.6.115.126:44389 requested a full tablet report, sending...
I20251210 14:59:57.756222 18557 ts_manager.cc:194] Registered new tserver with Master: 59f6512c9bcb407eb10d1d2abe7b22ce (127.6.115.65:35789)
I20251210 14:59:57.757678 18557 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:43665
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:57.804097 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:48861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:44389
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:57.937517 18780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:57.937803 18780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:57.937891 18780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:57.942540 18780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:57.942653 18780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:57.942790 18780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 14:59:57.948493 18780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.115.84:48861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:57.949934 18780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:57.950991 18780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:57.958613 18786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:57.958616 18787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:57.959007 18789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:57.959427 18780 server_base.cc:1047] running on GCE node
I20251210 14:59:57.959825 18780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:57.960408 18780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:57.961609 18780 hybrid_clock.cc:648] HybridClock initialized: now 1765378797961603 us; error 38 us; skew 500 ppm
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:57.964744 18780 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 14:59:57.965878 18780 webserver.cc:492] Webserver started at http://127.6.115.66:34309/ using document root <none> and password file <none>
I20251210 14:59:57.966527 18780 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:57.966588 18780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:57.966789 18780 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 14:59:57 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:57.980791 18780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "eab207ecb5d74e2ba7393023cbca36f5"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "c91b9ffce3f4d473d14377cf4388e7e5"
server_key_iv: "4b181787a4670ce78a9ffa6eb7ab7fd0"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.981276 18780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "eab207ecb5d74e2ba7393023cbca36f5"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "c91b9ffce3f4d473d14377cf4388e7e5"
server_key_iv: "4b181787a4670ce78a9ffa6eb7ab7fd0"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.984578 18780 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251210 14:59:57.997509 18796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:57.998684 18780 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251210 14:59:57.998811 18780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "eab207ecb5d74e2ba7393023cbca36f5"
format_stamp: "Formatted at 2025-12-10 14:59:57 on dist-test-slave-nb51"
server_key: "c91b9ffce3f4d473d14377cf4388e7e5"
server_key_iv: "4b181787a4670ce78a9ffa6eb7ab7fd0"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:57.998926 18780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:58.034507 18780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:58.038228 18780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:58.038493 18780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:58.039237 18780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:58.040355 18780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:58.040454 18780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:58.040548 18780 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:58.040634 18780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:58.054225 18780 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:45295
I20251210 14:59:58.055402 18780 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 14:59:58.057179 18909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:45295 every 8 connection(s)
I20251210 14:59:58.061622 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 18780
I20251210 14:59:58.061772 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 14:59:58.062067 6605 external_mini_cluster.cc:1468] Setting key e331b5d6c9defe59fb695de569a2cdcf
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:58.072929 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.059269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44389 (local address 127.6.115.66:45473)
1210 14:59:58.060407 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.060453 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.061510 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.062324 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.062334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.062688 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:58.063580 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.063597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.064654 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.064660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.065276 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.065289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.065592 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.066309 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:58.066342 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:58.067993 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:58.070208 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:58.070214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:58.070226 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:58.070538 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:58.070842 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:58.070845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:58.070847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:58.070960 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:58.071412 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:58.071419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:58.071669 (+ 250us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.071891 (+ 222us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.072144 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":115,"threads_started":1}
I20251210 14:59:58.073598 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.059015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.66:45473 (local address 127.6.115.126:44389)
1210 14:59:58.059172 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.059177 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.060594 (+ 1417us) server_negotiation.cc:408] Connection header received
1210 14:59:58.061772 (+ 1178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.061776 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.061841 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.061900 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:58.063759 (+ 1859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.064490 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.065454 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.065669 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.068192 (+ 2523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:58.068213 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:58.068216 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:58.068243 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:58.070070 (+ 1827us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.070666 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.070670 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.070672 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.070723 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.071091 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.071094 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.071097 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.071271 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:58.071359 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.073292 (+ 1933us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.073447 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251210 14:59:58.075158 18910 heartbeater.cc:344] Connected to a master server at 127.6.115.126:44389
I20251210 14:59:58.075397 18910 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:58.075981 18910 heartbeater.cc:507] Master 127.6.115.126:44389 requested a full tablet report, sending...
I20251210 14:59:58.077069 18557 ts_manager.cc:194] Registered new tserver with Master: eab207ecb5d74e2ba7393023cbca36f5 (127.6.115.66:45295)
I20251210 14:59:58.077776 18557 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.66@KRBTEST.COM'} at 127.6.115.66:45473
WARNING: no policy specified for kudu/127.6.115.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.67@KRBTEST.COM; defaulting to no policy
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 14:59:58.154594 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:48861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.67:0
--local_ip_for_outbound_sockets=127.6.115.67
--webserver_interface=127.6.115.67
--webserver_port=0
--tserver_master_addrs=127.6.115.126:44389
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 14:59:58.291426 18917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 14:59:58.291726 18917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 14:59:58.291824 18917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 14:59:58.296018 18917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 14:59:58.296144 18917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 14:59:58.296283 18917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.67
I20251210 14:59:58.301641 18917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.115.84:48861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.115.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.67
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 14:59:58.302739 18917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 14:59:58.303670 18917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 14:59:58.311161 18922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:58.311405 18917 server_base.cc:1047] running on GCE node
W20251210 14:59:58.311539 18923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 14:59:58.311187 18925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 14:59:58.312078 18917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 14:59:58.312625 18917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 14:59:58.313786 18917 hybrid_clock.cc:648] HybridClock initialized: now 1765378798313779 us; error 38 us; skew 500 ppm
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 14:59:58.317142 18917 init.cc:377] Logged in from keytab as kudu/127.6.115.67@KRBTEST.COM (short username kudu)
I20251210 14:59:58.318398 18917 webserver.cc:492] Webserver started at http://127.6.115.67:35203/ using document root <none> and password file <none>
I20251210 14:59:58.319106 18917 fs_manager.cc:362] Metadata directory not provided
I20251210 14:59:58.319216 18917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 14:59:58.319449 18917 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:58.333173 18917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data/instance:
uuid: "6cf1977247f6496889b7f044695ba3ba"
format_stamp: "Formatted at 2025-12-10 14:59:58 on dist-test-slave-nb51"
server_key: "ae021c11797aa0370fe3615c5725ed8c"
server_key_iv: "982031840a7091a3e1e05b51cc208427"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:58.333763 18917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance:
uuid: "6cf1977247f6496889b7f044695ba3ba"
format_stamp: "Formatted at 2025-12-10 14:59:58 on dist-test-slave-nb51"
server_key: "ae021c11797aa0370fe3615c5725ed8c"
server_key_iv: "982031840a7091a3e1e05b51cc208427"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:58.350925 18917 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.003s sys 0.000s
I20251210 14:59:58.365939 18932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:58.367280 18917 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251210 14:59:58.367388 18917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal
uuid: "6cf1977247f6496889b7f044695ba3ba"
format_stamp: "Formatted at 2025-12-10 14:59:58 on dist-test-slave-nb51"
server_key: "ae021c11797aa0370fe3615c5725ed8c"
server_key_iv: "982031840a7091a3e1e05b51cc208427"
server_key_version: "kuduclusterkey@0"
I20251210 14:59:58.367489 18917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 14:59:58.406886 18917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 14:59:58.410523 18917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 14:59:58.410768 18917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 14:59:58.411407 18917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 14:59:58.412585 18917 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 14:59:58.412681 18917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:58.412783 18917 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 14:59:58.412845 18917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 14:59:58.435587 18917 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.67:45173
I20251210 14:59:58.436026 19046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.67:45173 every 8 connection(s)
I20251210 14:59:58.436914 18917 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/data/info.pb
I20251210 14:59:58.443300 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 18917
I20251210 14:59:58.443445 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal/instance
I20251210 14:59:58.443807 6605 external_mini_cluster.cc:1468] Setting key 8428363b53508a1d25c94b767d0fc7a6
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.67@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 14:59:58.456305 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.440282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.67:42981 (local address 127.6.115.126:44389)
1210 14:59:58.440410 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.440414 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.440432 (+ 18us) server_negotiation.cc:408] Connection header received
1210 14:59:58.441057 (+ 625us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.441063 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.441125 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.441188 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:58.442995 (+ 1807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.443787 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.445297 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.445525 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.452030 (+ 6505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:58.452046 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:58.452049 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:58.452076 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:58.453983 (+ 1907us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.454582 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.454586 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.454588 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.454638 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.454999 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.455003 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.455004 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.455174 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:58.455242 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.456028 (+ 786us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.456152 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Dec 10 14:59:58 dist-test-slave-nb51 krb5kdc[18054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 14:59:58.458279 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.439266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44389 (local address 127.6.115.67:42981)
1210 14:59:58.439792 (+ 526us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.439839 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.440792 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.441631 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.441642 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.442063 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
1210 14:59:58.442813 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.442837 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.444247 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.444251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.445122 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.445134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.449537 (+ 4403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.450124 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 14:59:58.450156 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1210 14:59:58.451853 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 14:59:58.454128 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:58.454134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:58.454145 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:58.454458 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:58.454745 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 14:59:58.454748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 14:59:58.454750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 14:59:58.454874 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 14:59:58.455323 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 14:59:58.455329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 14:59:58.455599 (+ 270us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.456740 (+ 1141us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.456992 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20251210 14:59:58.459822 19047 heartbeater.cc:344] Connected to a master server at 127.6.115.126:44389
I20251210 14:59:58.460074 19047 heartbeater.cc:461] Registering TS with master...
I20251210 14:59:58.460669 19047 heartbeater.cc:507] Master 127.6.115.126:44389 requested a full tablet report, sending...
I20251210 14:59:58.461901 18557 ts_manager.cc:194] Registered new tserver with Master: 6cf1977247f6496889b7f044695ba3ba (127.6.115.67:45173)
I20251210 14:59:58.462278 6605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251210 14:59:58.462581 18557 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.67@KRBTEST.COM'} at 127.6.115.67:42981
I20251210 14:59:58.473657 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.464725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.6.115.126:44389)
1210 14:59:58.465006 (+ 281us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.465009 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.465429 (+ 420us) server_negotiation.cc:408] Connection header received
1210 14:59:58.465763 (+ 334us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.465765 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.465810 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.465876 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1210 14:59:58.467137 (+ 1261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.467599 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.468999 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.469183 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.470009 (+ 826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 14:59:58.470024 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 14:59:58.470028 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 14:59:58.470065 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1210 14:59:58.471895 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.472348 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.472352 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.472354 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.472399 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 14:59:58.472651 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 14:59:58.472655 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 14:59:58.472657 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 14:59:58.472812 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 14:59:58.472927 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.473391 (+ 464us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.473512 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20251210 14:59:58.476766 18557 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251210 14:59:58.479537 18557 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251210 14:59:58.495415 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.489076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35789 (local address 127.0.0.1:43562)
1210 14:59:58.491178 (+ 2102us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.491193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.491333 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.492092 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.492099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.492124 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:58.492426 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.492436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.494007 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.494011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.494936 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.494947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.495064 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.495112 (+ 48us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.495199 (+ 87us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.495279 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"thread_start_us":103,"threads_started":1}
I20251210 14:59:58.496435 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.488847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:45295 (local address 127.0.0.1:44792)
1210 14:59:58.489385 (+ 538us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.489421 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.490345 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.493583 (+ 3238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.493587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.493608 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:58.493871 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.493880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.495264 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.495267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.496089 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.496096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.496195 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.496214 (+ 19us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.496257 (+ 43us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.496310 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":89,"threads_started":1}
I20251210 14:59:58.497153 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.489629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.6.115.65:35789)
1210 14:59:58.490449 (+ 820us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.490456 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.491265 (+ 809us) server_negotiation.cc:408] Connection header received
1210 14:59:58.491556 (+ 291us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.491563 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.491741 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.491904 (+ 163us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:58.492582 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.493850 (+ 1268us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.495991 (+ 2141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.496719 (+ 728us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.496846 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.496910 (+ 64us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.496996 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":84,"threads_started":1}
I20251210 14:59:58.500188 18706 tablet_service.cc:1505] Processing CreateTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c (DEFAULT_TABLE table=test-table [id=d402f0f397734beab4575b42f2163eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:58.501232 18706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c27db4726884de2a01ea0bb9e17f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:58.501473 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.488956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.6.115.66:45295)
1210 14:59:58.490865 (+ 1909us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.490872 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.490893 (+ 21us) server_negotiation.cc:408] Connection header received
1210 14:59:58.490988 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.490993 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.492746 (+ 1753us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.492901 (+ 155us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:58.494016 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.495137 (+ 1121us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.497616 (+ 2479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.500521 (+ 2905us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.500871 (+ 350us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.501208 (+ 337us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.501308 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766,"thread_start_us":102,"threads_started":1}
I20251210 14:59:58.501616 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.490397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:45173 (local address 127.0.0.1:54670)
1210 14:59:58.491586 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.491601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.491870 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.497085 (+ 5215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.497088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.497108 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:58.497485 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.497496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.500104 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.500111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.501234 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.501243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.501347 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.501383 (+ 36us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.501435 (+ 52us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.501492 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":96,"threads_started":1}
I20251210 14:59:58.502626 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.490812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.6.115.67:45173)
1210 14:59:58.492073 (+ 1261us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.492079 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.492101 (+ 22us) server_negotiation.cc:408] Connection header received
1210 14:59:58.493138 (+ 1037us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.493143 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.496654 (+ 3511us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.497741 (+ 1087us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:58.498017 (+ 276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.499973 (+ 1956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.501776 (+ 1803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.502270 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.502358 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.502428 (+ 70us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.502501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":157,"threads_started":1}
I20251210 14:59:58.504040 18844 tablet_service.cc:1505] Processing CreateTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c (DEFAULT_TABLE table=test-table [id=d402f0f397734beab4575b42f2163eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:58.504877 18844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c27db4726884de2a01ea0bb9e17f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:58.504940 18971 tablet_service.cc:1505] Processing CreateTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c (DEFAULT_TABLE table=test-table [id=d402f0f397734beab4575b42f2163eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251210 14:59:58.505769 18971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c27db4726884de2a01ea0bb9e17f76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 14:59:58.513494 19067 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Bootstrap starting.
I20251210 14:59:58.513497 19068 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Bootstrap starting.
I20251210 14:59:58.515690 19068 tablet_bootstrap.cc:654] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:58.515690 19067 tablet_bootstrap.cc:654] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:58.516944 19068 log.cc:826] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:58.517069 19067 log.cc:826] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:58.517674 19066 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: Bootstrap starting.
I20251210 14:59:58.519726 19066 tablet_bootstrap.cc:654] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: Neither blocks nor log segments found. Creating new log.
I20251210 14:59:58.520129 19067 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: No bootstrap required, opened a new log
I20251210 14:59:58.520354 19067 ts_tablet_manager.cc:1403] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251210 14:59:58.520820 19068 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: No bootstrap required, opened a new log
I20251210 14:59:58.521034 19068 ts_tablet_manager.cc:1403] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20251210 14:59:58.523730 19067 raft_consensus.cc:359] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.523802 19068 raft_consensus.cc:359] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.523970 19068 raft_consensus.cc:385] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:58.524027 19068 raft_consensus.cc:740] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1977247f6496889b7f044695ba3ba, State: Initialized, Role: FOLLOWER
I20251210 14:59:58.524245 19067 raft_consensus.cc:385] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:58.524360 19067 raft_consensus.cc:740] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab207ecb5d74e2ba7393023cbca36f5, State: Initialized, Role: FOLLOWER
I20251210 14:59:58.524905 19067 consensus_queue.cc:260] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.525414 19066 log.cc:826] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: Log is configured to *not* fsync() on all Append() calls
I20251210 14:59:58.525916 19067 ts_tablet_manager.cc:1434] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251210 14:59:58.526446 19068 consensus_queue.cc:260] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.527038 18910 heartbeater.cc:499] Master 127.6.115.126:44389 was elected leader, sending a full tablet report...
I20251210 14:59:58.530815 19068 ts_tablet_manager.cc:1434] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20251210 14:59:58.531005 19066 tablet_bootstrap.cc:492] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: No bootstrap required, opened a new log
I20251210 14:59:58.531225 19066 ts_tablet_manager.cc:1403] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20251210 14:59:58.533701 19047 heartbeater.cc:499] Master 127.6.115.126:44389 was elected leader, sending a full tablet report...
I20251210 14:59:58.537473 19066 raft_consensus.cc:359] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.537732 19066 raft_consensus.cc:385] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 14:59:58.537914 19066 raft_consensus.cc:740] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f6512c9bcb407eb10d1d2abe7b22ce, State: Initialized, Role: FOLLOWER
I20251210 14:59:58.538801 19066 consensus_queue.cc:260] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.539402 18772 heartbeater.cc:499] Master 127.6.115.126:44389 was elected leader, sending a full tablet report...
I20251210 14:59:58.541296 19066 ts_tablet_manager.cc:1434] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251210 14:59:58.551445 19074 raft_consensus.cc:493] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251210 14:59:58.551654 19074 raft_consensus.cc:515] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.553004 19074 leader_election.cc:290] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eab207ecb5d74e2ba7393023cbca36f5 (127.6.115.66:45295), 6cf1977247f6496889b7f044695ba3ba (127.6.115.67:45173)
I20251210 14:59:58.557907 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.553554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:43473 (local address 127.6.115.66:45295)
1210 14:59:58.553676 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.553681 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.553693 (+ 12us) server_negotiation.cc:408] Connection header received
1210 14:59:58.553745 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.553748 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.553795 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.553870 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:58.554970 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.555682 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.556923 (+ 1241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.557450 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.557493 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.557709 (+ 216us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.557774 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251210 14:59:58.558638 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.553170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.66:45295 (local address 127.6.115.65:43473)
1210 14:59:58.553448 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.553464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.553557 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.554440 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.554443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.554465 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:58.554835 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.554842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.555832 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.555836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.556767 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.556778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.557556 (+ 778us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.557575 (+ 19us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.558420 (+ 845us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.558488 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20251210 14:59:58.559252 18911 tablet.cc:2378] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251210 14:59:58.559391 18849 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c27db4726884de2a01ea0bb9e17f76c" candidate_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab207ecb5d74e2ba7393023cbca36f5" is_pre_election: true
I20251210 14:59:58.559644 18849 raft_consensus.cc:2468] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f6512c9bcb407eb10d1d2abe7b22ce in term 0.
I20251210 14:59:58.560081 18660 leader_election.cc:304] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f6512c9bcb407eb10d1d2abe7b22ce, eab207ecb5d74e2ba7393023cbca36f5; no voters:
I20251210 14:59:58.560398 19074 raft_consensus.cc:2804] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251210 14:59:58.560469 19074 raft_consensus.cc:493] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251210 14:59:58.560514 19074 raft_consensus.cc:3060] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:58.561939 19074 raft_consensus.cc:515] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.562630 18849 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c27db4726884de2a01ea0bb9e17f76c" candidate_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab207ecb5d74e2ba7393023cbca36f5"
I20251210 14:59:58.562765 18849 raft_consensus.cc:3060] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:58.563701 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.554476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:34227 (local address 127.6.115.67:45173)
1210 14:59:58.554581 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.554585 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.559942 (+ 5357us) server_negotiation.cc:408] Connection header received
1210 14:59:58.560118 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.560122 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.560169 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.560216 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1210 14:59:58.560703 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.561483 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.562683 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.563204 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.563240 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.563494 (+ 254us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.563599 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251210 14:59:58.563979 18849 raft_consensus.cc:2468] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f6512c9bcb407eb10d1d2abe7b22ce in term 1.
I20251210 14:59:58.564065 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.554138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.67:45173 (local address 127.6.115.65:34227)
1210 14:59:58.559865 (+ 5727us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.559879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1210 14:59:58.560003 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 14:59:58.560331 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 14:59:58.560334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 14:59:58.560351 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1210 14:59:58.560544 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.560551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.561623 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.561627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 14:59:58.562543 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 14:59:58.562553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.563297 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.563312 (+ 15us) client_negotiation.cc:769] Sending connection context
1210 14:59:58.563910 (+ 598us) client_negotiation.cc:240] Negotiation successful
1210 14:59:58.563968 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5666,"thread_start_us":88,"threads_started":1}
I20251210 14:59:58.564522 18660 leader_election.cc:304] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f6512c9bcb407eb10d1d2abe7b22ce, eab207ecb5d74e2ba7393023cbca36f5; no voters:
I20251210 14:59:58.564579 18993 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c27db4726884de2a01ea0bb9e17f76c" candidate_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1977247f6496889b7f044695ba3ba" is_pre_election: true
I20251210 14:59:58.564822 18993 raft_consensus.cc:2468] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f6512c9bcb407eb10d1d2abe7b22ce in term 0.
I20251210 14:59:58.565277 18993 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c27db4726884de2a01ea0bb9e17f76c" candidate_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1977247f6496889b7f044695ba3ba"
I20251210 14:59:58.565395 18993 raft_consensus.cc:3060] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 0 FOLLOWER]: Advancing to term 1
I20251210 14:59:58.565487 19074 leader_election.cc:290] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [CANDIDATE]: Term 1 election: Requested vote from peers eab207ecb5d74e2ba7393023cbca36f5 (127.6.115.66:45295), 6cf1977247f6496889b7f044695ba3ba (127.6.115.67:45173)
I20251210 14:59:58.565622 19074 raft_consensus.cc:2804] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 1 FOLLOWER]: Leader election won for term 1
I20251210 14:59:58.565712 19074 raft_consensus.cc:697] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [term 1 LEADER]: Becoming Leader. State: Replica: 59f6512c9bcb407eb10d1d2abe7b22ce, State: Running, Role: LEADER
I20251210 14:59:58.566010 19074 consensus_queue.cc:237] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } }
I20251210 14:59:58.566498 18993 raft_consensus.cc:2468] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f6512c9bcb407eb10d1d2abe7b22ce in term 1.
I20251210 14:59:58.569114 18557 catalog_manager.cc:5654] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f6512c9bcb407eb10d1d2abe7b22ce (127.6.115.65). New cstate: current_term: 1 leader_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f6512c9bcb407eb10d1d2abe7b22ce" member_type: VOTER last_known_addr { host: "127.6.115.65" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 } health_report { overall_health: UNKNOWN } } }
I20251210 14:59:58.595932 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.591780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.6.115.65:35789)
1210 14:59:58.591937 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1210 14:59:58.591941 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1210 14:59:58.591954 (+ 13us) server_negotiation.cc:408] Connection header received
1210 14:59:58.592036 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 14:59:58.592038 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 14:59:58.592113 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 14:59:58.592205 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1210 14:59:58.592595 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.593331 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 14:59:58.594427 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 14:59:58.594638 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 14:59:58.594738 (+ 100us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1210 14:59:58.595310 (+ 572us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1210 14:59:58.595420 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1210 14:59:58.595771 (+ 351us) server_negotiation.cc:300] Negotiation successful
1210 14:59:58.595818 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20251210 14:59:58.607247 18849 raft_consensus.cc:1275] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 1 FOLLOWER]: Refusing update from remote peer 59f6512c9bcb407eb10d1d2abe7b22ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:59:58.607247 18993 raft_consensus.cc:1275] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 1 FOLLOWER]: Refusing update from remote peer 59f6512c9bcb407eb10d1d2abe7b22ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251210 14:59:58.607887 19074 consensus_queue.cc:1048] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab207ecb5d74e2ba7393023cbca36f5" member_type: VOTER last_known_addr { host: "127.6.115.66" port: 45295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:59:58.608042 19077 consensus_queue.cc:1048] T 6c27db4726884de2a01ea0bb9e17f76c P 59f6512c9bcb407eb10d1d2abe7b22ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf1977247f6496889b7f044695ba3ba" member_type: VOTER last_known_addr { host: "127.6.115.67" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251210 14:59:58.621248 19080 mvcc.cc:204] Tried to move back new op lower bound from 7230991559086821376 to 7230991558929928192. Current Snapshot: MvccSnapshot[applied={T|T < 7230991559086821376}]
I20251210 14:59:58.628165 19081 mvcc.cc:204] Tried to move back new op lower bound from 7230991559086821376 to 7230991558929928192. Current Snapshot: MvccSnapshot[applied={T|T < 7230991559086821376}]
I20251210 14:59:58.636119 18557 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:59:58.636263 18557 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40550:
table { table_name: "test-table" } modify_external_catalogs: true
I20251210 14:59:58.638757 18557 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 1e44d855812d4f86905718fa46550eb4: Sending DeleteTablet for 3 replicas of tablet 6c27db4726884de2a01ea0bb9e17f76c
I20251210 14:59:58.639428 18844 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:58 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:44792
I20251210 14:59:58.640324 18706 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:58 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:43562
I20251210 14:59:58.641149 18971 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c27db4726884de2a01ea0bb9e17f76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-10 14:59:58 UTC) from {username='kudu', principal='kudu/127.6.115.126@KRBTEST.COM'} at 127.0.0.1:54670
I20251210 14:59:58.641842 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 18639
I20251210 14:59:58.642639 19090 tablet_replica.cc:333] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: stopping tablet replica
I20251210 14:59:58.643170 19090 raft_consensus.cc:2243] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:59:58.643507 19090 raft_consensus.cc:2272] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:59:58.645130 19090 ts_tablet_manager.cc:1916] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:59:58.648324 19090 ts_tablet_manager.cc:1929] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:59:58.648470 19090 log.cc:1199] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-1/wal/wals/6c27db4726884de2a01ea0bb9e17f76c
I20251210 14:59:58.648792 19090 ts_tablet_manager.cc:1950] T 6c27db4726884de2a01ea0bb9e17f76c P eab207ecb5d74e2ba7393023cbca36f5: Deleting consensus metadata
I20251210 14:59:58.650028 18542 catalog_manager.cc:4985] TS eab207ecb5d74e2ba7393023cbca36f5 (127.6.115.66:45295): tablet 6c27db4726884de2a01ea0bb9e17f76c (table test-table [id=d402f0f397734beab4575b42f2163eb6]) successfully deleted
W20251210 14:59:58.652127 18543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.115.65:35789 (error 108)
I20251210 14:59:58.653595 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 14:59:58.653256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.65:35789 (local address 127.0.0.1:43590)
1210 14:59:58.653433 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1210 14:59:58.653510 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.65:35789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20251210 14:59:58.653788 18543 catalog_manager.cc:4712] TS 59f6512c9bcb407eb10d1d2abe7b22ce (127.6.115.65:35789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c27db4726884de2a01ea0bb9e17f76c: Network error: Client connection negotiation failed: client connection to 127.6.115.65:35789: connect: Connection refused (error 111)
I20251210 14:59:58.654091 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 18780
I20251210 14:59:58.656033 19092 tablet_replica.cc:333] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: stopping tablet replica
I20251210 14:59:58.656301 19092 raft_consensus.cc:2243] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20251210 14:59:58.656601 19092 raft_consensus.cc:2272] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251210 14:59:58.658095 19092 ts_tablet_manager.cc:1916] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251210 14:59:58.661160 19092 ts_tablet_manager.cc:1929] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251210 14:59:58.661294 19092 log.cc:1199] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Deleting WAL directory at /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765378657750705-6605-0/minicluster-data/ts-2/wal/wals/6c27db4726884de2a01ea0bb9e17f76c
I20251210 14:59:58.661688 19092 ts_tablet_manager.cc:1950] T 6c27db4726884de2a01ea0bb9e17f76c P 6cf1977247f6496889b7f044695ba3ba: Deleting consensus metadata
I20251210 14:59:58.662786 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 18917
I20251210 14:59:58.662938 18544 catalog_manager.cc:4985] TS 6cf1977247f6496889b7f044695ba3ba (127.6.115.67:45173): tablet 6c27db4726884de2a01ea0bb9e17f76c (table test-table [id=d402f0f397734beab4575b42f2163eb6]) successfully deleted
I20251210 14:59:58.668267 6605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskceAy_x/build/debug/bin/kudu with pid 18525
I20251210 14:59:58.676504 6605 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251210 14:59:59.010596 6605 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251210 14:59:59.010720 6605 mini_ranger.cc:66] Stopping Ranger...
I20251210 14:59:59.355089 6605 mini_ranger.cc:68] Stopped Ranger
2025-12-10 14:59:59.355 UTC [18085] LOG: received smart shutdown request
2025-12-10 14:59:59.358 UTC [18085] LOG: background worker "logical replication launcher" (PID 18093) exited with exit code 1
2025-12-10 14:59:59.358 UTC [18088] LOG: shutting down
2025-12-10 14:59:59.360 UTC [18088] LOG: checkpoint starting: shutdown immediate
2025-12-10 15:00:00.204 UTC [18088] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.034 s, sync=0.805 s, total=0.846 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/2045028, redo lsn=0/2045028
2025-12-10 15:00:00.213 UTC [18085] LOG: database system is shut down
2025-12-10T15:00:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45976 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 10 15:00:00 dist-test-slave-nb51 krb5kdc[19102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 10 15:00:00 dist-test-slave-nb51 krb5kdc[19102](info): set up 2 sockets
Dec 10 15:00:00 dist-test-slave-nb51 krb5kdc[19102](info): commencing operation
krb5kdc: starting...
W20251210 15:00:02.315081 6605 mini_kdc.cc:121] Time spent starting KDC: real 1.964s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/test-user.keytab.
Dec 10 15:00:02 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-10T15:00:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-10T15:00:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangeradmin_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangeradmin_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangerlookup_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangerlookup_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
I20251210 15:00:02.467195 6605 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskceAy_x/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/postgres -l logfile start
2025-12-10 15:00:04.812 UTC [19133] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-10 15:00:04.812 UTC [19133] LOG: listening on IPv4 address "127.6.115.84", port 58269
2025-12-10 15:00:04.815 UTC [19133] LOG: listening on Unix socket "/tmp/.s.PGSQL.58269"
2025-12-10 15:00:04.820 UTC [19138] LOG: database system was shut down at 2025-12-10 15:00:03 UTC
2025-12-10 15:00:04.823 UTC [19133] LOG: database system is ready to accept connections
I20251210 15:00:06.697503 6605 mini_postgres.cc:96] Postgres bound to 58269
2025-12-10 15:00:06.703 UTC [19144] FATAL: database "slave" does not exist
127.6.115.84:58269 - accepting connections
I20251210 15:00:06.704063 6605 mini_ranger.cc:161] Starting Ranger...
I20251210 15:00:06.721550 6605 mini_ranger.cc:84] Created miniranger Postgres user
I20251210 15:00:06.792214 6605 mini_ranger.cc:87] Created ranger Postgres database
I20251210 15:00:06.792310 6605 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin
2025-12-10 15:00:07,065 [I] DB FLAVOR :POSTGRES
2025-12-10 15:00:07,066 [I] --------- Verifying Ranger DB connection ---------
2025-12-10 15:00:07,066 [I] Checking connection..
2025-12-10 15:00:07,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-10 15:00:07,350 [I] Checking connection passed.
2025-12-10 15:00:07,350 [I] --------- Verifying version history table ---------
2025-12-10 15:00:07,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 15:00:07,637 [I] Table x_db_version_h does not exist in database ranger
2025-12-10 15:00:07,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 15:00:07,917 [I] Table x_db_version_h does not exist in database ranger
2025-12-10 15:00:07,917 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-10 15:00:07,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-10 15:00:08.233 UTC [19211] WARNING: there is no transaction in progress
2025-12-10 15:00:08,240 [I] create_dbversion_catalog.sql file imported successfully
2025-12-10 15:00:08,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-10 15:00:08,516 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-10 15:00:08,517 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-10 15:00:08,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-10 15:00:08,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-10 15:00:09,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nb51.c.gcp-upstream.internal','N') ;"
2025-12-10 15:00:09,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-10 15:00:09,624 [I] Table x_portal_user does not exist in database ranger
2025-12-10 15:00:09,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-10 15:00:09,903 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-10 15:00:09,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-10 15:00:10,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-10 15:00:10,508 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-10 15:00:10,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-10 15:00:11.288 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.299 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.310 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.320 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.331 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.341 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.382 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.386 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.395 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.404 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.412 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.420 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.427 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.431 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.435 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11.438 UTC [19328] WARNING: there is no transaction in progress
2025-12-10 15:00:11,825 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-10 15:00:11,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-10 15:00:12,102 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-10 15:00:12,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-10 15:00:12,402 [I] Table x_portal_user already exists in database 'ranger'
2025-12-10 15:00:12,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-10 15:00:12,711 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-10 15:00:12,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-10 15:00:12,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-10 15:00:13,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nb51.c.gcp-upstream.internal';"
2025-12-10 15:00:13,580 [I] CORE_DB_SCHEMA import status has been updated
2025-12-10 15:00:13,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-10 15:00:13,858 [I] DB_PATCHES have already been applied
I20251210 15:00:13.863826 6605 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskceAy_x/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251210 15:00:13.863888 6605 mini_ranger.cc:193] Using host: 127.6.115.84
I20251210 15:00:13.866756 6605 mini_ranger.cc:227] Ranger admin URL: http://127.6.115.84:44387
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-admin/ews/webapp].
Dec 10, 2025 3:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20251210 15:00:16.005460 6605 mini_ranger.cc:160] Time spent starting Ranger: real 9.301s user 0.004s sys 0.006s
Dec 10, 2025 3:00:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 10, 2025 3:00:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.MetricsREST
Dec 10, 2025 3:00:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 10, 2025 3:00:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 10, 2025 3:00:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20251210 15:00:34.986572 6605 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.115.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.115.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangerkms_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.115.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/rangerkms_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.115.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.115.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.115.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/HTTP_127.6.115.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 10 15:00:35 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378835, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251210 15:00:35.074579 6605 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251210 15:00:35.084626 6605 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251210 15:00:35.185808 6605 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251210 15:00:35.185904 6605 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms
I20251210 15:00:35.185922 6605 mini_ranger_kms.cc:227] Using postgres at 127.6.115.84:58269
2025-12-10 15:00:35,392 [I] DB FLAVOR :POSTGRES
2025-12-10 15:00:35,393 [I] --------- Verifying Ranger DB connection ---------
2025-12-10 15:00:35,393 [I] Checking connection
2025-12-10 15:00:35,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-10 15:00:35,736 [I] connection success
2025-12-10 15:00:35,736 [I] --------- Verifying Ranger DB tables ---------
2025-12-10 15:00:35,736 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-10 15:00:35,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-10 15:00:36,027 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-10 15:00:36,028 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-10 15:00:36,028 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-10 15:00:36,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.115.84:58269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-10 15:00:36,319 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251210 15:00:36.776120 6605 mini_ranger_kms.cc:310] Created kms service
I20251210 15:00:36.970693 6605 mini_ranger_kms.cc:326] Created kudu user
I20251210 15:00:37.012617 6605 mini_ranger_kms.cc:343] Created rangerkms user
Dec 10 15:00:37 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378835, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 15:00:37.388895 6605 mini_ranger_kms.cc:384] Added ranger policy
I20251210 15:00:37.389145 6605 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms:/tmp/dist-test-taskceAy_x/build/debug/bin/postgresql.jar:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskceAy_x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251210 15:00:37.389170 6605 mini_ranger_kms.cc:240] Using host: 127.6.115.84
I20251210 15:00:37.392412 6605 mini_ranger_kms.cc:276] Ranger KMS PID: 19520
I20251210 15:00:37.392493 6605 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.115.84:58345
Dec 10, 2025 3:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 10, 2025 3:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 10, 2025 3:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 10, 2025 3:00:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 10, 2025 3:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-kms/ews/webapp].
Dec 10, 2025 3:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20251210 15:00:39.645486 6605 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.571s user 0.000s sys 0.190s
I20251210 15:00:39.645689 6605 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251210 15:00:39.645776 6605 mini_ranger_kms.cc:401] 127.6.115.84:58345/kms/v1/keys
Dec 10 15:00:39 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765378839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Dec 10 15:00:40 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
Dec 10 15:00:40 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.115.84@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
Dec 10 15:00:42 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 15:00:42.492208 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:44721
--webserver_interface=127.6.115.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.115.126:44721
--ranger_config_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:58345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 15:00:42.601114 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 15:00:42.601320 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 15:00:42.601452 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 15:00:42.604553 19573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251210 15:00:42.604614 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 15:00:42.604653 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 15:00:42.604696 19573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251210 15:00:42.604734 19573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251210 15:00:42.608847 19573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.115.84:58345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.115.126:44721
--ranger_config_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.115.126:44721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.115.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 15:00:42.609848 19573 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 15:00:42.610618 19573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 15:00:42.616340 19579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 15:00:42.616467 19573 server_base.cc:1047] running on GCE node
W20251210 15:00:42.616361 19578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 15:00:42.616580 19581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 15:00:42.617128 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 15:00:42.618225 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 15:00:42.619441 19573 hybrid_clock.cc:648] HybridClock initialized: now 1765378842619417 us; error 43 us; skew 500 ppm
Dec 10 15:00:42 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 15:00:42.622483 19573 init.cc:377] Logged in from keytab as kudu/127.6.115.126@KRBTEST.COM (short username kudu)
I20251210 15:00:42.623736 19573 webserver.cc:492] Webserver started at http://127.6.115.126:33617/ using document root <none> and password file <none>
I20251210 15:00:42.624377 19573 fs_manager.cc:362] Metadata directory not provided
I20251210 15:00:42.624449 19573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 15:00:42.624687 19573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 15:00:42 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.126@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 15:00:42.674233 19573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data/instance:
uuid: "53cf58b312da436b9ae616bf1c713b7b"
format_stamp: "Formatted at 2025-12-10 15:00:42 on dist-test-slave-nb51"
server_key: "5c2ade5653842c5acb344ffcd6cea02a"
server_key_iv: "ddcc66beba1b305ef2661144902f91e2"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:42.674760 19573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal/instance:
uuid: "53cf58b312da436b9ae616bf1c713b7b"
format_stamp: "Formatted at 2025-12-10 15:00:42 on dist-test-slave-nb51"
server_key: "5c2ade5653842c5acb344ffcd6cea02a"
server_key_iv: "ddcc66beba1b305ef2661144902f91e2"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:42.678279 19573 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251210 15:00:42.694101 19588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:42.695189 19573 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251210 15:00:42.695297 19573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal
uuid: "53cf58b312da436b9ae616bf1c713b7b"
format_stamp: "Formatted at 2025-12-10 15:00:42 on dist-test-slave-nb51"
server_key: "5c2ade5653842c5acb344ffcd6cea02a"
server_key_iv: "ddcc66beba1b305ef2661144902f91e2"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:42.695405 19573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 15:00:42.712895 19573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 15:00:42.715791 19573 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 15:00:42.715968 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 15:00:42.723829 19573 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.126:44721
I20251210 15:00:42.723855 19640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.126:44721 every 8 connection(s)
I20251210 15:00:42.724823 19573 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/data/info.pb
I20251210 15:00:42.728098 19641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251210 15:00:42.728539 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 19573
I20251210 15:00:42.728670 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/wal/instance
I20251210 15:00:42.728915 6605 external_mini_cluster.cc:1468] Setting key 7600f47c79ae0670e11e65d6fce48a00
I20251210 15:00:42.734966 19641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b: Bootstrap starting.
Dec 10 15:00:42 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 15:00:42.737962 19641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b: Neither blocks nor log segments found. Creating new log.
I20251210 15:00:42.739154 19641 log.cc:826] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b: Log is configured to *not* fsync() on all Append() calls
I20251210 15:00:42.740926 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:42.730206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.6.115.126:44721)
1210 15:00:42.730582 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1210 15:00:42.730591 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1210 15:00:42.730624 (+ 33us) server_negotiation.cc:408] Connection header received
1210 15:00:42.731274 (+ 650us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 15:00:42.731290 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 15:00:42.731569 (+ 279us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 15:00:42.731889 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
1210 15:00:42.732793 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 15:00:42.733594 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 15:00:42.734236 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 15:00:42.734875 (+ 639us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 15:00:42.736756 (+ 1881us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 15:00:42.736795 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 15:00:42.736805 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 15:00:42.736838 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1210 15:00:42.738556 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 15:00:42.738967 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 15:00:42.738971 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 15:00:42.738975 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1210 15:00:42.739036 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 15:00:42.739339 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 15:00:42.739342 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 15:00:42.739344 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1210 15:00:42.739719 (+ 375us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 15:00:42.739938 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
1210 15:00:42.740149 (+ 211us) server_negotiation.cc:300] Negotiation successful
1210 15:00:42.740340 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20251210 15:00:42.741549 19641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b: No bootstrap required, opened a new log
I20251210 15:00:42.744586 19641 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } }
I20251210 15:00:42.744789 19641 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251210 15:00:42.744863 19641 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cf58b312da436b9ae616bf1c713b7b, State: Initialized, Role: FOLLOWER
I20251210 15:00:42.745283 19641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } }
I20251210 15:00:42.745440 19641 raft_consensus.cc:399] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251210 15:00:42.745499 19641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251210 15:00:42.745589 19641 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 0 FOLLOWER]: Advancing to term 1
I20251210 15:00:42.746526 19641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } }
I20251210 15:00:42.746866 19641 leader_election.cc:304] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53cf58b312da436b9ae616bf1c713b7b; no voters:
I20251210 15:00:42.747143 19641 leader_election.cc:290] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251210 15:00:42.747329 19646 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 1 FOLLOWER]: Leader election won for term 1
I20251210 15:00:42.747560 19646 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [term 1 LEADER]: Becoming Leader. State: Replica: 53cf58b312da436b9ae616bf1c713b7b, State: Running, Role: LEADER
I20251210 15:00:42.747887 19646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } }
I20251210 15:00:42.748211 19641 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [sys.catalog]: configured and running, proceeding with master startup.
I20251210 15:00:42.748865 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53cf58b312da436b9ae616bf1c713b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } } }
I20251210 15:00:42.748965 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [sys.catalog]: This master's current role is: LEADER
I20251210 15:00:42.749151 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53cf58b312da436b9ae616bf1c713b7b. Latest consensus state: current_term: 1 leader_uuid: "53cf58b312da436b9ae616bf1c713b7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf58b312da436b9ae616bf1c713b7b" member_type: VOTER last_known_addr { host: "127.6.115.126" port: 44721 } } }
I20251210 15:00:42.749233 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b [sys.catalog]: This master's current role is: LEADER
I20251210 15:00:42.749517 19653 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251210 15:00:42.752765 19653 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251210 15:00:42.753197 19641 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251210 15:00:42.763417 19653 catalog_manager.cc:1357] Generated new cluster ID: 52bf4dcb73a5441481b22ce88d6f864a
I20251210 15:00:42.763572 19653 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251210 15:00:42.771854 19653 catalog_manager.cc:1380] Generated new certificate authority record
I20251210 15:00:42.772991 19653 catalog_manager.cc:1514] Loading token signing keys...
I20251210 15:00:42.779435 19653 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 53cf58b312da436b9ae616bf1c713b7b: Generated new TSK 0
I20251210 15:00:42.780251 19653 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 15:00:43.471868 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:58345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.65:0
--local_ip_for_outbound_sockets=127.6.115.65
--webserver_interface=127.6.115.65
--webserver_port=0
--tserver_master_addrs=127.6.115.126:44721
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251210 15:00:43.579074 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 15:00:43.579272 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 15:00:43.579357 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 15:00:43.582620 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 15:00:43.582696 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 15:00:43.582799 19687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.65
I20251210 15:00:43.587172 19687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.115.84:58345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.115.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:44721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.65
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 15:00:43.588163 19687 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 15:00:43.588971 19687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 15:00:43.596547 19694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 15:00:43.596747 19697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 15:00:43.596592 19695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 15:00:43.597057 19687 server_base.cc:1047] running on GCE node
I20251210 15:00:43.597347 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 15:00:43.597872 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 15:00:43.599020 19687 hybrid_clock.cc:648] HybridClock initialized: now 1765378843599013 us; error 47 us; skew 500 ppm
Dec 10 15:00:43 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 15:00:43.601500 19687 init.cc:377] Logged in from keytab as kudu/127.6.115.65@KRBTEST.COM (short username kudu)
I20251210 15:00:43.602411 19687 webserver.cc:492] Webserver started at http://127.6.115.65:36587/ using document root <none> and password file <none>
I20251210 15:00:43.602911 19687 fs_manager.cc:362] Metadata directory not provided
I20251210 15:00:43.602957 19687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 15:00:43.603116 19687 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 15:00:43 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 15:00:43.617070 19687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data/instance:
uuid: "9a2ce639a7c444d8b4b6015577c089ea"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8bf8b11dbda091ebefaebe13672ceae2"
server_key_iv: "fc2ebaf36c6b9e2a96c6bcd87b7381e1"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.617697 19687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2ce639a7c444d8b4b6015577c089ea"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8bf8b11dbda091ebefaebe13672ceae2"
server_key_iv: "fc2ebaf36c6b9e2a96c6bcd87b7381e1"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.622591 19687 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20251210 15:00:43.637269 19704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.638656 19687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251210 15:00:43.638767 19687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal
uuid: "9a2ce639a7c444d8b4b6015577c089ea"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8bf8b11dbda091ebefaebe13672ceae2"
server_key_iv: "fc2ebaf36c6b9e2a96c6bcd87b7381e1"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.638870 19687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 15:00:43.662338 19687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 15:00:43.665881 19687 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 15:00:43.666128 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 15:00:43.666851 19687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 15:00:43.667901 19687 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 15:00:43.668004 19687 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.668130 19687 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 15:00:43.668213 19687 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.685667 19687 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.65:32995
I20251210 15:00:43.686906 19687 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/data/info.pb
I20251210 15:00:43.689898 19817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.65:32995 every 8 connection(s)
I20251210 15:00:43.694687 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 19687
I20251210 15:00:43.694818 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-0/wal/instance
I20251210 15:00:43.695097 6605 external_mini_cluster.cc:1468] Setting key a1d29b37978abbc1c58494394d06c0c8
Dec 10 15:00:43 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.65@KRBTEST.COM for kudu/127.6.115.126@KRBTEST.COM
I20251210 15:00:43.705360 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:43.691739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.115.65:57915 (local address 127.6.115.126:44721)
1210 15:00:43.692030 (+ 291us) server_negotiation.cc:207] Beginning negotiation
1210 15:00:43.692035 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1210 15:00:43.694009 (+ 1974us) server_negotiation.cc:408] Connection header received
1210 15:00:43.695015 (+ 1006us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1210 15:00:43.695019 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1210 15:00:43.695080 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1210 15:00:43.695144 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1210 15:00:43.696836 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 15:00:43.697715 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1210 15:00:43.699059 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1210 15:00:43.699271 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 15:00:43.700969 (+ 1698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1210 15:00:43.700987 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1210 15:00:43.700989 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1210 15:00:43.701011 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1210 15:00:43.703129 (+ 2118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 15:00:43.703726 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 15:00:43.703733 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 15:00:43.703736 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 15:00:43.703799 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1210 15:00:43.704187 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1210 15:00:43.704191 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1210 15:00:43.704194 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1210 15:00:43.704368 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1210 15:00:43.704456 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1210 15:00:43.705063 (+ 607us) server_negotiation.cc:300] Negotiation successful
1210 15:00:43.705181 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20251210 15:00:43.706099 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:43.691931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:57915)
1210 15:00:43.693858 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:43.693893 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1210 15:00:43.694753 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 15:00:43.695557 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1210 15:00:43.695567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1210 15:00:43.695918 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
1210 15:00:43.696663 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 15:00:43.696681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 15:00:43.697859 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1210 15:00:43.697866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1210 15:00:43.698485 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1210 15:00:43.698499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1210 15:00:43.698748 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1210 15:00:43.699377 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1210 15:00:43.699416 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1210 15:00:43.700831 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1210 15:00:43.703273 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 15:00:43.703279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 15:00:43.703290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1210 15:00:43.703599 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 15:00:43.703913 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1210 15:00:43.703917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1210 15:00:43.703919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1210 15:00:43.704059 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1210 15:00:43.704508 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1210 15:00:43.704515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1210 15:00:43.704802 (+ 287us) client_negotiation.cc:769] Sending connection context
1210 15:00:43.705027 (+ 225us) client_negotiation.cc:240] Negotiation successful
1210 15:00:43.705257 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.6.115.66@KRBTEST.COM; defaulting to no policy
I20251210 15:00:43.707383 19818 heartbeater.cc:344] Connected to a master server at 127.6.115.126:44721
I20251210 15:00:43.707657 19818 heartbeater.cc:461] Registering TS with master...
I20251210 15:00:43.708209 19818 heartbeater.cc:507] Master 127.6.115.126:44721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.115.66@KRBTEST.COM" created.
I20251210 15:00:43.709838 19605 ts_manager.cc:194] Registered new tserver with Master: 9a2ce639a7c444d8b4b6015577c089ea (127.6.115.65:32995)
I20251210 15:00:43.711366 19605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.115.65@KRBTEST.COM'} at 127.6.115.65:57915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.6.115.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.115.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.6.115.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab.
I20251210 15:00:43.760505 6605 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskceAy_x/build/debug/bin/kudu
/tmp/dist-test-taskceAy_x/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.115.84:58345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.115.66:0
--local_ip_for_outbound_sockets=127.6.115.66
--webserver_interface=127.6.115.66
--webserver_port=0
--tserver_master_addrs=127.6.115.126:44721
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.6.115.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.6.115.126 from keytab /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20251210 15:00:43.780503 19679 server.cc:273] Received an EOF from the subprocess
F20251210 15:00:43.781417 19666 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fc53734bdcd google::LogMessage::Fail() at ??:0
@ 0x7fc53734fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc53734b7cc google::LogMessage::Flush() at ??:0
@ 0x7fc53734cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5390022d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fc538fff0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fc539003005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x565488a9aa0a std::function<>::operator()() at ??:0
@ 0x7fc5387a57dc kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc53903e6db start_thread at ??:0
@ 0x7fc53650271f clone at ??:0
W20251210 15:00:43.897173 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251210 15:00:43.897459 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251210 15:00:43.897536 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251210 15:00:43.901546 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251210 15:00:43.901602 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251210 15:00:43.901676 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.115.66
I20251210 15:00:43.906570 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.115.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.115.84:58345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/krb5kdc/kudu.keytab
--principal=kudu/127.6.115.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.115.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.115.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.6.115.126:44721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.115.66
--log_dir=/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 10 Dec 2025 14:43:15 UTC on 24a791456cd2
build id 9446
I20251210 15:00:43.907614 19828 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251210 15:00:43.908495 19828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251210 15:00:43.915786 19845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 15:00:43.915796 19846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251210 15:00:43.915871 19848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251210 15:00:43.916618 19828 server_base.cc:1047] running on GCE node
I20251210 15:00:43.917032 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251210 15:00:43.917659 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251210 15:00:43.918835 19828 hybrid_clock.cc:648] HybridClock initialized: now 1765378843918819 us; error 27 us; skew 500 ppm
Dec 10 15:00:43 dist-test-slave-nb51 krb5kdc[19102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251210 15:00:43.922026 19828 init.cc:377] Logged in from keytab as kudu/127.6.115.66@KRBTEST.COM (short username kudu)
I20251210 15:00:43.923286 19828 webserver.cc:492] Webserver started at http://127.6.115.66:36905/ using document root <none> and password file <none>
I20251210 15:00:43.923942 19828 fs_manager.cc:362] Metadata directory not provided
I20251210 15:00:43.924023 19828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251210 15:00:43.924265 19828 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 10 15:00:43 dist-test-slave-nb51 krb5kdc[19102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765378843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.115.66@KRBTEST.COM for HTTP/127.6.115.84@KRBTEST.COM
I20251210 15:00:43.939275 19828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data/instance:
uuid: "d6c3101b07ba4e63b66def57cb40da57"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8e5db8bc66bc387ec2c6fe7220649384"
server_key_iv: "471741f08de90b9a92fcf89d2824c951"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.939735 19828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance:
uuid: "d6c3101b07ba4e63b66def57cb40da57"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8e5db8bc66bc387ec2c6fe7220649384"
server_key_iv: "471741f08de90b9a92fcf89d2824c951"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.943416 19828 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251210 15:00:43.958413 19855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.959568 19828 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20251210 15:00:43.959690 19828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data,/tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal
uuid: "d6c3101b07ba4e63b66def57cb40da57"
format_stamp: "Formatted at 2025-12-10 15:00:43 on dist-test-slave-nb51"
server_key: "8e5db8bc66bc387ec2c6fe7220649384"
server_key_iv: "471741f08de90b9a92fcf89d2824c951"
server_key_version: "kuduclusterkey@0"
I20251210 15:00:43.959793 19828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251210 15:00:43.971242 19828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251210 15:00:43.973847 19828 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251210 15:00:43.974015 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251210 15:00:43.974530 19828 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251210 15:00:43.975373 19828 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251210 15:00:43.975419 19828 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.975466 19828 ts_tablet_manager.cc:616] Registered 0 tablets
I20251210 15:00:43.975498 19828 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251210 15:00:43.986299 19828 rpc_server.cc:307] RPC server started. Bound to: 127.6.115.66:36027
I20251210 15:00:43.986761 19968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.115.66:36027 every 8 connection(s)
I20251210 15:00:43.987402 19828 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/data/info.pb
I20251210 15:00:43.996949 6605 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskceAy_x/build/debug/bin/kudu as pid 19828
I20251210 15:00:43.997072 6605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskceAy_x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1765378657750705-6605-0/minicluster-data/ts-1/wal/instance
I20251210 15:00:43.997296 6605 external_mini_cluster.cc:1468] Setting key a47792964c961254e8ecd4580a4eb9ae
W20251210 15:00:44.024742 19100 connection.cc:537] client connection to 127.6.115.126:44721 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251210 15:00:44.025775 19971 negotiation.cc:337] Failed RPC negotiation. Trace:
1210 15:00:43.989447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:48227)
1210 15:00:43.989882 (+ 435us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:43.989924 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1210 15:00:43.990627 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1210 15:00:44.025062 (+ 34435us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
W20251210 15:00:44.026314 19969 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:44721 (0 consecutive failures): Network error: Failed to ping master at 127.6.115.126:44721: Client connection negotiation failed: client connection to 127.6.115.126:44721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251210 15:00:44.026985 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:44.026730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:54399)
1210 15:00:44.026879 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:44.026935 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20251210 15:00:44.027624 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:44.027443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:42053)
1210 15:00:44.027541 (+ 98us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:44.027578 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251210 15:00:44.027778 19969 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251210 15:00:44.714200 19818 heartbeater.cc:499] Master 127.6.115.126:44721 was elected leader, sending a full tablet report...
I20251210 15:00:44.715076 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:44.714656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:56845)
1210 15:00:44.714913 (+ 257us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:44.714978 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20251210 15:00:44.715337 19818 heartbeater.cc:646] Failed to heartbeat to 127.6.115.126:44721 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
I20251210 15:00:45.028904 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:45.028419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:48495)
1210 15:00:45.028733 (+ 314us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:45.028792 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20251210 15:00:45.716363 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:45.715965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:50417)
1210 15:00:45.716205 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:45.716253 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20251210 15:00:46.030110 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:46.029712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:47671)
1210 15:00:46.029949 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:46.030001 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20251210 15:00:46.717617 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:46.717129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:51765)
1210 15:00:46.717470 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:46.717523 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20251210 15:00:47.031298 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:47.030857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:45059)
1210 15:00:47.031130 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:47.031205 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20251210 15:00:47.718829 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:47.718368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:44497)
1210 15:00:47.718672 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:47.718731 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20251210 15:00:48.032553 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:48.032114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:60175)
1210 15:00:48.032398 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:48.032456 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20251210 15:00:48.720175 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:48.719724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.65:34833)
1210 15:00:48.720018 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:48.720079 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20251210 15:00:49.033998 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1210 15:00:49.033495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.115.126:44721 (local address 127.6.115.66:47257)
1210 15:00:49.033834 (+ 339us) negotiation.cc:107] Waiting for socket to connect
1210 15:00:49.033897 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.115.126:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
F20251210 15:00:49.062016 6605 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1765378849 (unix time) try "date -d @1765378849" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000019cd) received by PID 6605 (TID 0x7f29c21add80) from PID 6605; stack trace: ***
@ 0x7f29cde49980 (unknown) at ??:0
@ 0x7f29cb480fb7 gsignal at ??:0
@ 0x7f29cb482921 abort at ??:0
@ 0x7f29cc6eedcd google::LogMessage::Fail() at ??:0
@ 0x7f29cc6f2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f29cc6ee7cc google::LogMessage::Flush() at ??:0
@ 0x7f29cc6eff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29cf47acab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f29cf4779c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x560b5ac9e92f kudu::SecurityITest::StartCluster() at ??:0
@ 0x560b5ac86c15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f29cd6fe0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f29cd6f2bdb testing::Test::Run() at ??:0
@ 0x7f29cd6f2d9d testing::TestInfo::Run() at ??:0
@ 0x7f29cd6f3377 testing::TestSuite::Run() at ??:0
@ 0x7f29cd6f377c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f29cd6fe60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f29cd6f2e63 testing::UnitTest::Run() at ??:0
@ 0x560b5acd6c7e RUN_ALL_TESTS() at ??:0
@ 0x560b5acd5e6b main at ??:0
@ 0x7f29cb463bf7 __libc_start_main at ??:0
@ 0x560b5ac6e06a _start at ??:0
2025-12-10 15:00:49.239 UTC [19551] FATAL: terminating connection due to unexpected postmaster exit
2025-12-10 15:00:49.240 UTC [19447] FATAL: terminating connection due to unexpected postmaster exit
2025-12-10 15:00:49.242 UTC [19448] FATAL: terminating connection due to unexpected postmaster exit
2025-12-10 15:00:49.243 UTC [19444] FATAL: terminating connection due to unexpected postmaster exit
2025-12-10 15:00:49.245 UTC [19445] FATAL: terminating connection due to unexpected postmaster exit
2025-12-10 15:00:49.246 UTC [19443] FATAL: terminating connection due to unexpected postmaster exit